fix(evals): make the live board hold up in a real terminal

MI
4d ago
+1384-195

Stack

Stacked on #3164 (evals/tui-followups). Review against that parent, not main.

Summary

The live board from #3090 fell apart in a real terminal: repeated status bars, half-drawn frames, raw [stagehand] ERROR … lines cut mid-word, 102 "failures" after pressing Esc. I drove the real CLI end to end in a PTY (node-pty rendered through a headless xterm, against Browserbase) and fixed what that showed. Before the fix, 516 of 527 sampled screens of evals run act were corrupted; after it, every scenario shows one status bar and no stray lines.

Board corruption

  • The 4.x SDK writes its log lines straight to process.stderr (and also hands them to onLog). Only console.* was captured, so those writes landed between redraws and every frame after them was drawn in the wrong place. Direct stdout/stderr writes are now captured while the board is live (or with -v): lines inside a row go to that row's log, the SDK's raw copy is dropped, and the board's own writes pass through.
  • Line-per-event mode (CI, pipes) is unchanged: raw writes still pass through there.

Stopping

  • Esc stops after the rows in flight; a second Esc stops them now. Same behaviour in argv mode and the REPL (argv mode used to exit on the first Esc with no summary).
  • Rows that never ran, or were killed by the hard stop, are stopped: not failures, not in the pass rate. The summary always prints, headed "stopped after N of M runs".
  • After a hard stop, a row still running 3 s after its session closed is abandoned. Tasks can retry against a dead session for minutes; in one run they held "stop now" open past a 200 s timeout.

Log stream (v, -v, --follow)

  • Lines name the case and, when the run has several, the model.
  • The SDK's per-step info lines (stagehand.init, Act inference completed, …) stay in the row's log file; warnings and errors stream.
  • A line a row repeats in a burst prints once with ↳ repeated N more times. Lines are cut to one screen line.
  • One-row mode marks the followed row with ▸, names it in the hint, and moves to the next row when it finishes.
  • ? is a short block, one line per key.

Failures and summary

  • The board's recent list shows the reason first; a session URL only when it fits whole (it used to be cut mid-UUID, hiding the reason).
  • A task's own message counts as the reason, and a failed row's log ends with it.
  • The summary groups failures by cause, largest first, and points credential errors at the doctor:
  act · 120 runs · 2m35s

  cell                                   pass
  stagehand × anthropic/claude-haiku-4-5   5% 2/40
  stagehand × openai/gpt-4.1-mini         85% 34/40
  stagehand × google/gemini-2.5-flash     78% 31/40

  Failures (53)
  ✗ 36×   fail       invalid x-api-key  claude-haiku-4-5  → check the key: evals doctor --probe
  ✗ 8×    fail       task check failed (no reason given)  3 models · act/iframe_form_filling ×2, act/vantechjournal ×2…
  ✗ 3×    fail       unable to click on the button  gpt-4.1-mini, gemini-2.5-flash · act/csr_in_oopif ×2, act/osr_in_o…
  ✗ act/custom_dropdown gemini-2.5-flash  fail       unable to expand the dropdown
      https://www.browserbase.com/sessions/…
  • Count columns appear only when some cell uses them; bench runs no longer print the row-by-row table under -v; the header, help and summary fit an 80-column terminal.

Test plan

  • vitest packages/evals/tests: 1253 passing (the three Playwright-backed integration files need a local browser)
  • tsc, oxlint (0 errors), fmt:check
  • Every new test fails with its fix reverted (26 mutation checks: stream capture, raw SDK lines, stopped rows, hard-stop grace, Esc staging, reason-first lines, followed row, repeat collapse, header wrap, credential grouping, …)
  • End to end in a PTY against Browserbase, 120-run act matrix at -c 50: full run with v/? toggles, single Esc, double Esc, 80×24 terminal, and the REPL (stop, second run, exit). One status bar and no raw lines in each; double Esc exits in ~3 s.
  • README screenshots not regenerated: run-live.png, run-verbose.png and run-summary.png still show the old recent list and summary.

Summary by cubic

Fixes the evals live board so it renders correctly in a real terminal: the Stagehand 4.x SDK writes its log lines straight to process.stderr, landing between the board's redraws — 516 of 527 sampled screens of evals run act were corrupted. Direct stdout/stderr writes are now captured while the board is live, so every frame holds up.

Stopping and results

  • Esc stops after the rows in flight; a second Esc stops them now, in argv mode and the REPL alike.
  • Rows the user stopped count as stopped — not failures, not in the pass rate, and not in the verifier, gate or failure-kind counts, so a killed row's verifier error can't trip the dead-judge exit; the summary always prints.
  • Repeated hard stops share one cleanup; rows are released only after sessions close, the grace timer clears when a row settles, and a session that starts after the sweep is cleaned up on registration.
  • A row aborted while queued for its retry says "stopped before its retry"; one that already retried still counts in retried.
  • Failures are grouped by cause, largest first, with one session URL per group; credential errors point at evals doctor --probe, and a task's own message counts as its failure reason.
  • Count columns appear only when used; header, help and summary fit an 80-column terminal.

Log stream (v, -v, --follow)

  • Lines name the case (and model when the run has several) and are cut to one screen line; repeated lines print once with a count, and partial writes buffer per row.
  • Line-per-event mode marks stopped rows with their own glyph and counts them in the progress bar.
  • The SDK's per-step info stays in the row's log file; one-row mode marks the followed row with ▸ and moves on when it finishes.