Shell: handle CRLF, tab, comment, and backslash-newline line endings in the lexer

RO
1mo ago
+583-32

Fixes #29669

Problem

  • A .sh file with CRLF line endings fails under bun run ./repro.sh: bun: command not found: export and error: Script not found "build\r". Every Windows-edited script has this shape.
  • The shell lexer (src/shell_parser/parse.rs) has arms for \n, space, and #, but none for \r or \t. A \r fell through to append_char_to_str_pool and was glued onto the word before it.

Fix

  • Add a \r arm: in Normal state, \r\n is one newline. Inside quotes the byte stays literal, as in bash and dash. A lone \r stays literal too.
  • Treat \<CR><LF> like \<LF>: a line continuation in Normal and Double state, removed before tokenizing (POSIX 2.2.1). It joins the halves of a word and of a variable name ($FOO\<LF>ls is $FOOls, as in bash). Double state now lets a backslash escape \r. A bare \<CR> in double quotes keeps its backslash.
  • Sibling bugs in the same code: an unquoted tab breaks words like a space. A # comment ends at its newline (a trailing backslash does not continue it) and emits the Newline token, so the next line is a new command. The parser skips newlines after |, &&, and || (POSIX linebreak), so cmd | on one line and the next command on the following line parse. Any escaped char ends a variable name: $HOME\foo is ${HOME}foo. A # starts a comment wherever a new token starts, so echo a;# note no longer runs a command named #. Inside backticks a comment ends at the closing backtick.
  • Verified: test/cli/run/run-shell.test.ts (10 new tests, all fail on main) and test/js/bun/shell/lex.test.ts (token-level cases for CRLF, lone CR, continuations, tab, comments, plus six Bun.$ cases). Also ran parse.test.ts and bunshell.test.ts.

Background

  • The lexer is one match on the next char. An arm either does break 'escaped (append the char to the current word) or sets fell_through (skip it). break_word flushes the current word as a Text token. Newline and Semicolon tokens separate statements in the parser.
  • read_char marks a char as escaped when a backslash precedes it. In Double state only a short list of chars can be escaped ($, backtick, ", \, \n, #). \r joins that list so \<CR><LF> can be a continuation in double quotes.
  • eat_var scans a variable name after $. It now skips an escaped LF or CRLF and keeps scanning, and stops at any other escaped char, so the name matches what bash reads after it removes \<newline>. The Bun.$ interpolation guard (outbuf_ends_with_var_ref in shell_body.rs) looks past a trailing \<newline> for the same reason, so $FOO\<LF>${x} expands FOO and keeps x a separate value.
  • Considered normalizing CRLF to LF when run_command.rs reads the file: that leaves Bun.$ template strings and quoted CR bytes wrong. The lexer is the one place every shell source passes through.

Downsides

  • A Bun.$ template or script that relied on \t being part of a word, on $FOO\<LF>ls expanding as ${FOO}ls, or on $HOME\foo expanding HOMEfoo, changes meaning. All now match bash and dash.
  • None found for input without \r or \t beyond one table lookup per such byte: both were already in SPECIAL_CHARS, so the fast-path scanner stopped on them before this change too.
Notes

Behaviour compared against bash 5 and dash for every new test input. FOO=x; FOOls=y; echo "[$FOO\<LF>ls]" [$FOO\<LF>ls] [$\<LF>FOO] prints [y] [y] [x] in bash, dash, and this branch.

Earlier iterations of this PR patched the Zig lexer. Main replaced that file with the Rust lexer in src/shell_parser/parse.rs and the fix was ported.

The continuation-join test uses the echo builtin instead of printf: Windows has no printf, and the external one on CI printed n for \n.

Pre-existing and left alone: echo ** foo glues foo onto ** (break_word pushes no Delimit after DoubleAsterisk). Stock bun reports no matches found: **foo.

Operator continuation checked with echo a | # c, cat, echo b &&, echo c, false || # d, echo e on separate lines: bash, dash, and this branch print a b c e. Stock bun fails with expected a command or assignment.

no test proof · iteration 17 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/cli/run/run-shell.test.ts