ci(docs): check the japanese docs against deepl

Draft
KA
13m ago
+1041-0

What and why

Gives every translated docs section a second opinion from DeepL, so a mistranslated or stale section gets flagged in review.

  • scripts/check-docs-translations-deepl.ts splits docs/ja/<path>.mdx and docs/<path>.mdx at their headings, which pair up by position because check-docs-translations.ts keeps the outlines equal. For each section a pull request changes, in either language, DeepL translates our Japanese back into English and the English into Japanese.
  • scripts/deepl-review.ts scores both with chrF (character n-grams, so no Japanese tokenizer). A section is flagged when it is far from DeepL in both directions, left in English, or missing a glossary term that DeepL's version uses. A correct translation that only words things differently usually stays close in one direction.
  • The table in docs/ja/README.md is uploaded as a DeepL glossary (EN→JA and JA→EN), so DeepL uses our terms (ホスト型ブラウザ, サーバー). It is reused by a content hash, so an unchanged glossary costs nothing.
  • A section is flagged when both scores are below 0.25. That cut-off comes from the first real run on the site's UI copy: the one real mistranslation scored 0.11 at best, while rewordings scored up to 0.44.
  • A flag is a warning on the section's heading plus a side-by-side row in the job summary (English, DeepL's English, ours, DeepL's Japanese, scores). Every reviewed section, with its scores, is also uploaded as the deepl-report artifact, since GitHub shows only 10 warnings per step. The summary ends with the month's DeepL usage.
  • It never fails the job on a flag: DeepL is a second opinion, not the source of truth. A used-up monthly quota skips the review with a warning. The job fails only when DeepL can't be reached.
  • .github/workflows/docs-translations.yml runs it on pull requests that touch the docs. It is advisory and not a required check. It needs a DEEPL_API_KEY secret (Free plan, :fx key); without one, as on a fork, it skips.
  • Only changed sections are sent. The whole Japanese docs are about 1M characters, which is the Free plan's whole month. A full quickstart is about 8k.
  • Fallback. When DeepL's monthly quota runs out, the same review continues with openai/gpt-6-luna on the Vercel AI Gateway (AI_GATEWAY_API_KEY, already a repo secret). The model gets the glossary and the です・ます style in its prompt and answers a JSON list, one translation per section. The report and warnings name whoever translated. Without a gateway key, a used-up quota still skips the review with a warning.

Stacked on #1875. e2e-inc/e2e-site checks its UI copy the same way.

Issue

ENG-1360

Verified

Ran it locally: yes

  • Fallback, live: with DeepL stubbed to answer 456, gpt-6-luna reviewed the 9 quickstart sections in 23 s. Scores ranged from 0.65 to 1.00, and the only flags were 2 glossary ones.
  • node --test scripts/deepl-review.test.ts: chrF, glossary parsing (including the real README's 35 terms), the review rules, MDX section splitting, the real quickstart pairing up section by section, and the DeepL client (glossary created once and then reused, request shape, a 456 error surfaced).
  • Ran the script against a stubbed DeepL: warnings land on docs/ja/quickstart.mdx at each section's heading line, and --base HEAD sends nothing.
  • Ran against the real DeepL API with the repo's key, over every translated section (1,191, before it was limited to changed sections). It worked end to end, and used up the month's quota.
  • A used-up quota, stubbed with a 456, ends the job with a warning and exit 0.
  • pnpm docs:check, oxlint scripts/, and tsc -p scripts pass.