Derive the stringWidth tables from the UCD and share one cluster width with sliceAnsi and console.table

RO
2mo ago
+1274-1262

Problem

  • Bun.stringWidth disagrees with the Unicode properties and with string-width on 3,993 of the 1,112,064 scalars, and on common emoji sequences. Examples: "\u20E3" is 2, "🇦" is 1 but "🇦\u0301" is 2, "😀🏻" is 4, "\uFFF9" is 1, 126 unassigned Indic codepoints are 0.
  • The cause is isZeroWidth() in scripts/generate-stringwidth-tables.mjs, a hand-kept range list, plus grapheme classes copied from a Unicode 15.1 table (671 InCB=Consonant and 689 Extended_Pictographic entries differ from 17.0). GraphemeState::width() in stringWidth.cpp sets the keycap flag on a lone U+20E3 and treats an emoji modifier as a cluster break.
  • console.table and the markdown renderer size cells with visibleUTF8Width, which summed code points with no clustering: "👨‍👩‍👧" is 2 in stringWidth and 6 in a table cell. sliceAnsi.cpp carried its own copy of the cluster width rules.

Fix

  • The generator derives every table field from the UCD 17.0 files (EastAsianWidth, DerivedGeneralCategory, DerivedCoreProperties, GraphemeBreakProperty, emoji-data). Zero-width is Cc, Cf, Mn, Me, Cs, unassigned Default_Ignorable, Hangul V/T jamo, and Indic Mc (the kept convention). Wide is East Asian Width W/F. A Control grapheme class adds GB4/GB5.
  • GraphemeState moves to stringWidth.h and is the one cluster width for stringWidth, sliceAnsi and the UTF-8 path. A keycap needs a [0-9#*] base. A lone regional indicator stays 1 wide, with or without a combining mark after it (string-width agrees). An emoji modifier extends any base (GB9) and never widens it. © and ® are emoji bases, so the firstCp special case is gone.
  • visibleUTF8WidthExcludeANSI and utf8IndexAtWidthExcludeANSI walk grapheme clusters through the same accumulator as the UTF-16 path. wrapAnsi's hard wrap advances by cluster too, so a flag or ZWJ sequence is never split across rows.
  • Verified: test/js/bun/util/stringWidth.test.ts (two new describe blocks, 14 tests fail on 1.4.3), sliceAnsi.test.ts, wrapAnsi.test.ts (two new hard-wrap tests, 44 expectations regenerated, every row checked against columns). Also sliceAnsi-fuzz, stripANSI, wrapAnsi.npm, console-table, bun-inspect-table, test/js/bun/md, markdown-entrypoint, repl: 2260 pass.

Background

  • The width table is a 3-stage lookup from a codepoint to one packed byte: grapheme break class, width class (zero, narrow, wide, ambiguous) and the Emoji bit. stringWidthTables.h is generated. Run bun scripts/generate-stringwidth-tables.mjs to rebuild it.
  • A grapheme cluster is the unit a terminal draws as one glyph (UAX #29). The width code sums the widths of complete clusters. GraphemeState accumulates one cluster and applies the emoji rules (flag pair, keycap, skin tone, ZWJ sequence, VS15/VS16) on top of the sum.
  • Kept conventions that differ from node's per-code-point width: an Indic consonant plus vowel sign is one column, a digit plus VS16 is one column, halfwidth katakana plus a voiced sound mark is two columns.
Notes

Sweep against the property-derived reference model (node 26.3, ICU 78.3, Unicode 17.0) over all 1,112,064 scalars: 1.4.3 agrees on 99.641%, this branch on 99.987%. The 143 that remain:

  • The 26 regional indicators are 1, the reference (Emoji_Presentation) and node say 2. Kept at 1 on purpose: string-width returns 1, #25447 pinned it, and the existing tests assert it. The fix here is only that a mark after a lone one no longer turns it into 2.

  • 101 Indic spacing vowel signs (Mc) are 0, the reference says 1. Kept on purpose so "\u0915\u093F" stays one column. glibc 2.41 wcwidth() returns 1 for them. A maintainer can flip this by removing one line in isZeroWidth().

  • U+115F, U+3164 (Hangul fillers, 2) and U+FFA0 (1): default-ignorable but East Asian Wide letters. node returns 2, 2, 1.

  • 8 unassigned codepoints in the Hangul Jamo Extended-B block: the reference zero-widths the whole block.

  • Kirat Rai vowel signs (Unicode 16) have Grapheme_Cluster_Break=V but are letters, so the jamo rule is bounded to cp <= U+D7FF.

Sequence sweep (2,707 base x modifier combinations): the remaining differences are the kept conventions above, plus sequences where the reference's own rule is loose (a digit plus ZWJ, a combining mark as the cluster base).

wrapAnsi.test.ts: the 44 changed rows all involve " \u20E3" (space plus keycap, now one column instead of two) or "\u0600 👍🏿" (prepend plus a modifier sequence, now 2 columns instead of 4). A script re-evaluated only the expected column of the table with the new build and asserted Bun.stringWidth(row) <= columns for every row. The family emoji hard-wrap test changes from one codepoint per row to one row, because a hard wrap now moves whole clusters.

Self-review: 4 concerns raised, 4 addressed (cluster-aware wrapWord, a Prepend before a bulk-counted unit, SIMD ASCII counting on the width-only UTF-8 path, the test table regeneration).

The UCD files are not reachable from the CI network. The generator accepts --ucd <dir> with the five files by base name.

Bun__isEmojiPresentation is removed. sliceAnsi.cpp was its only user, and it now reads the Emoji bit through StringWidth::fusedClassify.

[human-review] gate passed · iteration 2 · 9 files touched

fails on main (without fix) 
ASAN without fix: 59 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/util/sliceAnsi.test.ts test/js/bun/util/stringWidth.test.ts test/js/bun/util/wrapAnsi.test.ts
bun test v1.4.3 (f42e98025)

test/js/bun/util/stringWidth.test.ts:
(pass) stringWidth [217.22ms]
(pass) toMatchNPMStringWidth > ansi colors [103.29ms]
(pass) toMatchNPMStringWidthExcludeANSI > ansi colors [46.70ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [5.18ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [3.78ms]
(pass) upstream [16.43ms]
(pass) upstream [17.41ms]
(pass) ambiguousIsNarrow=false [38.99ms]
(pass) ignores control characters [16.37ms]
(pass) ignores control characters [6.20ms]
(pass) handles combining characters [4.25ms]
(pass) handles combining characters [1.75ms]
(pass) handles ZWJ characters [10.88ms]
(pass) handles ZWJ characters [6.32ms]
(pass) stringWidth extended > zero-width characters > soft hyphen (U+00AD) [4.55ms]
(pass) stringWidth extended > zero-width characters > word joiner and invisible operators (U+2060-U+2064) [8.08ms]
(pass) stringWidth extended > zer
... (truncated)

release without fix: 59 FAILED
bun test v1.4.3-canary.1 (f42e98025)

test/js/bun/util/stringWidth.test.ts:
(pass) stringWidth [10.90ms]
(pass) toMatchNPMStringWidth > ansi colors [1.83ms]
(pass) toMatchNPMStringWidthExcludeANSI > ansi colors [2.63ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [0.26ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [0.10ms]
(pass) upstream [0.54ms]
(pass) upstream [4.55ms]
(pass) ambiguousIsNarrow=false [1.86ms]
(pass) ignores control characters [0.22ms]
(pass) ignores control characters [0.10ms]
(pass) handles combining characters [0.05ms]
(pass) handles combining characters [0.02ms]
(pass) handles ZWJ characters [0.13ms]
(pass) handles ZWJ characters [0.08ms]
(pass) stringWidth extended > zero-width characters > soft hyphen (U+00AD) [0.76ms]
(pass) stringWidth extended > zero-width characters > word joiner and invisible operators (U+2060-U+2064) [0.06ms]
(pass) stringWidth extended > zero-width characters > zero-width space/joiner/non-joiner (U+200B-U+200D) [0.04ms]
(pass) stringWidth extended > zero-width characters > LRM and RLM (U+200E-U+200F) [0.03ms]
(pass) stringWidth extended > zero-width characters > bidi embeddi
... (truncated)
passes on PR (with fix) 
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/util/sliceAnsi.test.ts test/js/bun/util/stringWidth.test.ts test/js/bun/util/wrapAnsi.test.ts
bun test v1.4.3 (f42e98025)

test/js/bun/util/stringWidth.test.ts:
(pass) stringWidth [187.37ms]
(pass) toMatchNPMStringWidth > ansi colors [50.22ms]
(pass) toMatchNPMStringWidthExcludeANSI > ansi colors [46.36ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [5.19ms]
(pass) leading non-ansi characters in UTF-16 string seems to fail [3.04ms]
(pass) upstream [16.94ms]
(pass) upstream [13.02ms]
(pass) ambiguousIsNarrow=false [17.16ms]
(pass) ignores control characters [6.12ms]
(pass) ignores control characters [3.33ms]
(pass) handles combining characters [2.42ms]
(pass) handles combining characters [0.62ms]
(pass) handles ZWJ characters [5.13ms]
(pass) handles ZWJ characters [3.39ms]
(pass) stringWidth extended > zero-width characters > soft hyphen (U+00AD) [2.78ms]
(pass) stringWidth extended > zero-width characters > word joiner and invisible operators (U+2060-U+2064) [3.50ms]
(pass) stringWidth extended > zero-w
... (truncated)

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 5552ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[1/10] gen cpp.rs (cppbind)
[2/9] cxx obj/unified/UnifiedSource-src_jsc_bindings-5.cpp.o
[3/9] cxx obj/unified/UnifiedSource-src_jsc_bindings-1.cpp.o
[4/9] link bun-profile
rust-lld: warning: Linking two modules of different target triples: 'obj/codegen/GeneratedSSLConfig.cpp.o' is 'x86_64-pc-linux-gnu' whereas 'rust-target/x86_64-unknown-linux-gnu/release/libbun_runtime.a(bun_jsc-e420352fc50a33cc.bun_jsc.782f39e5b0322f0c-cgu.0.rcgu.o at 80120150)' is 'x86_64-unknown-linux-gnu'


rust-lld: warning: Linking two modules of different target triples: 'obj/codegen/GeneratedSocketConfig.cpp.o' is 'x86_64-pc-linux-gnu' whereas 'rust-target/x86_64-unknown-linux-gnu/release/libbun_runtime.a(bun_jsc-e420352fc50a33cc.bun_jsc.782f39e5b0322f0c-cgu.0.rcgu.o at 80120150)' is 'x86_64-unknown-linux-gnu'


rust-lld: warning: Linking two modules of different target triples: 'obj/codegen/GeneratedSocketConfigHandlers.cpp.o' is 'x86_64-pc-linux-gnu' whereas 'rust-target/x86_64-unknown-linux-gnu/release/libbun_runtime.a(bun_js
... (truncated)
diff hotspot 
scripts/generate-stringwidth-tables.mjs |  251 ++++---
 src/jsc/bindings/sliceAnsi.cpp          |   86 +--
 src/jsc/bindings/stringWidth.cpp        |  544 ++++++---------
 src/jsc/bindings/stringWidth.h          |  134 +++-
 src/jsc/bindings/stringWidthTables.h    | 1133 +++++++++++++------------------
 src/jsc/bindings/wrapAnsi.cpp           |   58 +-
 test/js/bun/util/sliceAnsi.test.ts      |    8 +-
 test/js/bun/util/stringWidth.test.ts    |  203 +++++-
 test/js/bun/util/wrapAnsi.test.ts       |  119 ++--
 9 files changed, 1274 insertions(+), 1262 deletions(-)

gate history · 1 passed · 0 rejected · iteration 2

evidence per changed file 
file                                     reads  edits  tests
scripts/generate-stringwidth-tables.mjs      1      3     45
src/jsc/bindings/sliceAnsi.cpp               0      0     44
src/jsc/bindings/stringWidth.cpp             3      0     45
src/jsc/bindings/stringWidth.h               1      2     43
src/jsc/bindings/stringWidthTables.h         1      0     45
src/jsc/bindings/wrapAnsi.cpp                1      0     43
test/js/bun/util/sliceAnsi.test.ts           1      2     13
test/js/bun/util/stringWidth.test.ts         3      5     31
test/js/bun/util/wrapAnsi.test.ts            0      0     21