feat: Bun shell

Merged
ZA
3y ago
+19283-854

What does this PR do?

Note that this is currently still a WIP, but opening this now to make reviewing easier because the changes are approaching 15k lines. I'm sorry.

This adds a cross-platform bash-like shell scripting language to Bun. Scripts are runnable through a zx inspired API that allows for embedding of JS expressions into the shell command (similar to embedding JS expressions in JSX):

import { $ } from 'bun'

const output = new Uint8Array(1 << 20);
// Redirect the output of `ls` to output
await $`ls > ${output}`

Bun shell scripts will also be runnable in package.json scripts, or through bun run, and files with the appropriate shebang.

Shell features

  • Supports ascii and unicode
  • Pipelining cmd1 | cmd2
  • Conditional control flow cmd1 && cmd2
    • &&
    • ||
  • Redirections (cmd > lmao.txt)
    • stdout/stderr (foo 1> lmao.txt or foo 2> lmao.txt)
    • stdin
  • Shell variables
    • Assigning FOO=bar
    • Variable expansion echo $FOO
    • Local variables (only used by shell for expansion)
    • Exported (used in env)
  • Command substitution (echo $(cmd with args))
  • Brace expansion echo {a,b,c}{d,e,f}
    • Nested braces echo {a,b,{c,d}}{e,f}
  • Glob expansion
  • zx inspired JS api
    • Redirecting to JS objects
      • Array buffer
      • Subprocesses?
      • Other shell instances?
    • JS objects in a pipeline? (e.g. ${someBuffer} | cmd1 | cmd2)
    • Custom thenable to lazily execute scripts
  • Runnable through bun run
  • Runnable through package.json script
  • Shebang

Supported builtin commands

  • export: set environment variables for the shell session to be used by subprocesses
  • cd: change the current working directory for the shell
    • cd <arg>
    • cd - (go to previous directory)
  • echo: write text to stdout
  • pwd: print previous directory
  • which: print absolute paths of stuff
  • rm: delete stuff
    • recursively delete directories concurrently
    • -v: verbose mode, prints out what is deleted
    • -i: interactive mode, ask user for confirmation
  • mv: move/rename stuff
    • move files and directories
    • verbose mode
    • support moving across different filesystems
    • interactive mode
  • ls: list stuff in directories
    • recursive mode
    • -a/-A: show all or show everything except . and ..
    • all the other flags
  • cp
  • mkdir
  • cat
  • sleep

Other stuff todo

  • Fix error handling in some places
  • Builtin commands need to write UTF-16 text to stdout/stderr on windows I think
  • Write more tests
  • Write tests for memory leaks
  • JS template arguments are currently .protect()ed and .unprotect()ed, but this is slow need a better way to do it
  • Shell escape template argument strings
  • Would be cool to support command substitution like this: 
    const firstCmd = $`ls`
await $`mv $(${firstCmd}) new_dir/`

Reviewing notes

Lexer/parser

The src/shell.zig file contains the lexer and parser for the shell language. This file also contains the v1 blocking and synchronous interpreter which isn't really used right now but might be used for a sync API.

Shell subprocess

The src/shell/subproc.zig file contains a modified version of the Subprocess struct for the shell. Originally, it was supposed to share some code with the original Subprocess, (which was inside src/shell/util.zig) but it became a bit too complicated to abstract it for both because they had different requirements so now they are separate things and src/shell/util.zig should probable be deleted.

Interpreter

The src/shell/interpreter.zig file contains the interpreter rewritten to be non-blocking. It also contains the implementation for all the builtin shell commands.

The interpreter works by walking the AST and creating state machine structs for each node.

For example, for something like cmd1 && cmd2 it would create a state machine for the binary expression, storing some state like whether its executing the left hand side or the right hand side. It will also create state machines for its children, cmd1 and cmd2 which need their own state for all the execution state of managing the subprocess.

When we need to do work that might block like writing to a file descriptor, we schedule the poll and stop doing work in the interpreter, basically suspending execution. Then when the poll is ready it will call a function on whichever state machine it was associated with and resume execution

Memory management

The lexer/parser allocate tokens/ast nodes into an arena. The AST and arena are both handed to the Interpreter.

The interpreter has to heap allocate state machines, so it uses an allocator different from the AST arena to do that.

Builtin commands, when executed, need to store their argv and some environment variables, those are allocated into an arena. The builtin has access to that arena for the duration of its execution if it wants to use it for quick scratch allocations. For example, when the builtin fails it needs to write the error message to stderr so it might use the arena to allocate a formatted string