fix(ci): cover Unicode Tag block + other invisibles in check-unicode-safety (ASCII smuggling)

Merged
JA
5mo ago
+93-1

Summary

scripts/ci/check-unicode-safety.js is the repo's last line of defence before contributor content reaches agent context. It also runs in --write auto-sanitize mode on .md / .mdx / .txt. Found two classes of invisible code points the denylist silently passes through.

1. Unicode Tag block (U+E0000–U+E007F)

Tag characters were proposed for language tagging in Unicode 3.1 and have been deprecated since Unicode 5.1, so no legitimate text uses them. They are the canonical vector for "ASCII Smuggling" / "Tag Smuggling" LLM prompt injection, demonstrated against multiple LLM assistants during 2024–2025: an attacker hides instructions inside an ASCII-looking string (PR body, SKILL.md, frontmatter), the model reads the tag bytes, the human reviewer sees nothing.

2. Other widely-cited invisibles the denylist also missed

Code pointNameNotes
U+180EMONGOLIAN VOWEL SEPARATORReclassified Zs → Cf in Unicode 6.3; renders zero-width
U+115FHANGUL CHOSEONG FILLERZero-width Korean shaping filler
U+1160HANGUL JUNGSEONG FILLERZero-width Korean shaping filler
U+2061–U+2064FUNCTION APPLICATION, INVISIBLE TIMES/SEPARATOR/PLUSMath typesetting only
U+3164HANGUL FILLERDiscord / Twitter smuggling references

Reproduction (before this PR)

$ mkdir -p /tmp/uni-test && node -e "
    const fs = require('fs');
    const hidden = [...Array(5)].map((_,i) => String.fromCodePoint(0xE0041 + i)).join('');
    fs.writeFileSync('/tmp/uni-test/innocent.md',
      '# Title\n\nBenign text' + hidden + ' more.\n');"

$ ECC_UNICODE_SCAN_ROOT=/tmp/uni-test node scripts/ci/check-unicode-safety.js
Unicode safety check passed.
$ echo $?
0

Expected: tag-block characters reported as dangerous-invisible violations (exit 1) and stripped under --write.
Actual: validator passes; --write leaves the bytes intact.

After this PR

$ ECC_UNICODE_SCAN_ROOT=/tmp/uni-test node scripts/ci/check-unicode-safety.js
Unicode safety violations detected:
innocent.md:3:12 dangerous-invisible U+E0041
innocent.md:3:14 dangerous-invisible U+E0042
innocent.md:3:16 dangerous-invisible U+E0043
innocent.md:3:18 dangerous-invisible U+E0044
innocent.md:3:20 dangerous-invisible U+E0045
exit=1

--write mode strips the bytes (file length 47 → 42, regex /[\u{E0000}-\u{E007F}]/u no longer matches).

Commit layout

Three reviewable commits:

  1. fix(ci): cover Unicode Tag block (U+E0000–U+E007F) in check-unicode-safety — single new range in isDangerousInvisibleCodePoint. Primary security fix.
  2. fix(ci): cover other widely-cited invisible code points in check-unicode-safety — five additional code points / ranges (U+180E, U+115F, U+1160, U+2061–U+2064, U+3164). Each documented with its specific abuse pattern.
  3. test(ci): regression coverage for newly-covered invisible code points — 9 new test cases (detection for each newly-covered code point + a --write integration test that strips Tag block + U+180E from a mixed-content fixture).

Tests

  • tests/scripts/check-unicode-safety.test.js: 5 → 14 cases
  • Full yarn test green; yarn lint clean
  • ECC self-scan (node scripts/ci/check-unicode-safety.js) reports only the pre-existing U+2605 BLACK STAR warnings (unchanged) — no new in-repo violations introduced.

Risk

Very low. The change is purely additive to a denylist; no legitimate text uses any of these code points. Reviewer concern would be "are we sure?" — answer for the Tag block is deprecation since Unicode 5.1; the other invisibles are zero-width and either obsolete (Mongolian VS) or math-only (U+2061–U+2064) or Korean shaping fillers that should not appear in source / markdown / PR bodies.

Test plan

  • CI lint passes
  • CI tests pass
  • Manual: tag-block fixture reproduction exits 1 with 5 violations
  • Manual: --write mode on the same fixture strips bytes and leaves surrounding text intact
  • Manual: ECC self-scan still exits with the same status (only pre-existing emoji warnings)

Summary by cubic

Blocks Unicode Tag block and other invisible code points in the Unicode safety check to prevent ASCII/tag smuggling. Detection now fails on these, and --write strips them.

  • Bug Fixes
    • Denylist adds U+E0000–U+E007F (Tag block), U+180E, U+115F, U+1160, U+2061–U+2064, U+3164.
    • Violations reported as dangerous-invisible; --write sanitizes .md/.mdx/.txt.
    • Regression tests increased from 5 to 14, including a write-mode integration test; no new in-repo violations.

Summary by CodeRabbit

Release Notes

  • Chores
    • Improved Unicode safety checks to detect and sanitize a broader range of invisible Unicode characters that could affect content integrity.

Review Change Stack