ci(docs): check the japanese docs against deepl
Draft
KA
13m ago+1041+1041+0123456789012345678901234567890123456789-0-0-0123456789
What and why
Gives every translated docs section a second opinion from DeepL, so a mistranslated or stale section gets flagged in review.
scripts/check-docs-translations-deepl.tssplitsdocs/ja/<path>.mdxanddocs/<path>.mdxat their headings, which pair up by position becausecheck-docs-translations.tskeeps the outlines equal. For each section a pull request changes, in either language, DeepL translates our Japanese back into English and the English into Japanese.scripts/deepl-review.tsscores both with chrF (character n-grams, so no Japanese tokenizer). A section is flagged when it is far from DeepL in both directions, left in English, or missing a glossary term that DeepL's version uses. A correct translation that only words things differently usually stays close in one direction.- The table in
docs/ja/README.mdis uploaded as a DeepL glossary (EN→JA and JA→EN), so DeepL uses our terms (ホスト型ブラウザ, サーバー). It is reused by a content hash, so an unchanged glossary costs nothing. - A section is flagged when both scores are below 0.25. That cut-off comes from the first real run on the site's UI copy: the one real mistranslation scored 0.11 at best, while rewordings scored up to 0.44.
- A flag is a warning on the section's heading plus a side-by-side row in the job summary (English, DeepL's English, ours, DeepL's Japanese, scores). Every reviewed section, with its scores, is also uploaded as the
deepl-reportartifact, since GitHub shows only 10 warnings per step. The summary ends with the month's DeepL usage. - It never fails the job on a flag: DeepL is a second opinion, not the source of truth. A used-up monthly quota skips the review with a warning. The job fails only when DeepL can't be reached.
.github/workflows/docs-translations.ymlruns it on pull requests that touch the docs. It is advisory and not a required check. It needs aDEEPL_API_KEYsecret (Free plan,:fxkey); without one, as on a fork, it skips.- Only changed sections are sent. The whole Japanese docs are about 1M characters, which is the Free plan's whole month. A full quickstart is about 8k.
- Fallback. When DeepL's monthly quota runs out, the same review continues with
openai/gpt-6-lunaon the Vercel AI Gateway (AI_GATEWAY_API_KEY, already a repo secret). The model gets the glossary and the です・ます style in its prompt and answers a JSON list, one translation per section. The report and warnings name whoever translated. Without a gateway key, a used-up quota still skips the review with a warning.
Stacked on #1875. e2e-inc/e2e-site checks its UI copy the same way.
Issue
ENG-1360
Verified
Ran it locally: yes
- Fallback, live: with DeepL stubbed to answer 456,
gpt-6-lunareviewed the 9 quickstart sections in 23 s. Scores ranged from 0.65 to 1.00, and the only flags were 2 glossary ones. node --test scripts/deepl-review.test.ts: chrF, glossary parsing (including the real README's 35 terms), the review rules, MDX section splitting, the real quickstart pairing up section by section, and the DeepL client (glossary created once and then reused, request shape, a 456 error surfaced).- Ran the script against a stubbed DeepL: warnings land on
docs/ja/quickstart.mdxat each section's heading line, and--base HEADsends nothing. - Ran against the real DeepL API with the repo's key, over every translated section (1,191, before it was limited to changed sections). It worked end to end, and used up the month's quota.
- A used-up quota, stubbed with a 456, ends the job with a warning and exit 0.
pnpm docs:check,oxlint scripts/, andtsc -p scriptspass.