journeys: the action fuzzer (correctness and speed) as a mode of the smoke-gate driver

Merged
LA
5d ago
+17993-0

Summary

The action fuzzer for cmux NEXT, built as a mode of the journeys / dogfood smoke-gate driver (cmux-ad 2026-10-31: "we need to fuzz greater for both correctness and speed"; fuzz-catalog rows a and m). One action layer and one set of oracles with the smoke gate.

  • Brings scripts/journeys (the journeys runner and the dogfood smoke gate, from cp/pa-d-dogfood-smoke-gate at 1222849c09f, owner agent a7031071e77d3e582) to feat-cmux-next. Today it lives only on perf/chrome-parity and that branch.
  • scripts/journeys/ext_fuzz.py: step fuzz_run. It plays a seeded plan of user actions through debug.key (the key router's real path), ext_window's drags and the driver's relaunch, at mixed timings (now, pair within 60 ms, slow), with a model of workspaces, tabs, terminals and panes. Oracles after each burst: create (or a visible refusal), r3a and r3b through ext_smoke's own cond_smoke_r3, _widths and _columns, focus, topology (daemon == app), stall (> 100 ms in cmux code), crash, and leak after close-all. A failure is shrunk in the same app with ddmin. Speed marks per create: shown_ms and typeable_ms.
  • scripts/journeys/fuzz.py: rounds through run.py (exclusive, pool perf-lead, never cmux17), a --base A/B on the same mini (ABBA), confirmation of each shrunk repro in a fresh lease, written as a regression journey. Verdicts: CLEAN, RED, NOT JUDGED, DIAGNOSTIC. A .clean file is written only for a judged clean run.
  • Action sets: layout (row m: New Column, Cmd-D, Split Down, Ctrl-, close pane, divider drag, window resize, relaunch) and full`.
  • journeys/fuzz/: the fuzz journey, and Lawrence's reported bugs as fixed sequences (lawrence-*.json).

This is not a CI job, step or gate (Lawrence 2026-10-23). It runs on the fleet, started by the lead, the tip watcher and the dogfood flow.

Testing

  • python3 scripts/journeys/test_fuzz.py: 16 OK (plan, bursts, ddmin, the speed table, and the verdicts with a stand-in run.py).
  • python3 scripts/journeys/test_fuzz_sim.py: 8 OK. The oracles run against a simulated app. A correct app plays clean. Each planted bug fails with its own oracle (second split lost, column resize, grid mismatch, PTY mismatch, a stall in cmux code, a PTY leak), and ddmin shrinks it.
  • python3 scripts/journeys/test_smoke_gate.py: OK. run.py --check: 59 valid.
  • python3 scripts/cmux-next/prepush-gate.py: no new violations.

Observed / Not run

  • Observed: the Python tests above, on the agent VM.
  • Fleet runs (diagnostic, exclusive perf-lead leases): the layout set on feat-tip-fixes-v2 (f2f3083bac80ab71101d03ac), v2.2.11-opt, v2.2.12-opt (152ca95f959989d8d6c8c250) and v2.2.10-opt. It found an R3b red (New Column narrows the old column 872 -> 291 pt, feat-tip-fixes-v2) and an R3a red (a stale fit_grid after Cmd-D, on v2.2.10-opt and v2.2.12-opt). Both are saved as red journeys in journeys/fuzz/found-*.json. feat-tip-fixes-v1 cannot be driven: it is a plain Release build with no debug.* methods.
  • Not run: whole-host (key window) mode. It needs the Cua grant on cmux17.

Performance

No perf numbers: this changes only harness Python under scripts/journeys (no app, daemon or web code). The fuzzer's own speed marks (shown_ms and typeable_ms for each create, an A/B against --base) are results it reports about the builds it tests. They are not a performance change of this PR.

Changelog

none

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Adds a model-based action fuzzer for cmux NEXT as a mode of the journeys smoke-gate driver, bringing the journeys runner and smoke gate from cp/pa-d-dogfood-smoke-gate with the fuzzer built on top.

The fuzzer (ext_fuzz.py)

  • Plays seeded plans of user actions through debug.key, ext_window's drags and the driver's relaunch at mixed timings, with oracles for create, keys, R3a/R3b, focus, daemon-vs-app topology, stall, crash and leak.
  • The create oracle counts new ids, not net counts, so a close landing after the baseline never hides a create.
  • A split_type burst runs Cmd-D (or Split Down) with a line typed at once, then a second split 30 ms later with its own line; each must land in its own new pane or it is a keys red (TIP RED 1193).
  • On builds with debug.creation.hold/release, the hold is proven by the release reply, so n(hold) = n(rounds); a split the app refused with a visible message is not judged.
  • A fuzz_crash_reports step classifies a mini's crash reports as product or harness.

The orchestrator (fuzz.py)

  • Rounds run through run.py on exclusive perf-lead leases (cmux19 only by default), with an optional --base ABBA; every lease waits on the mini scheduler's go hand-off, where one GO is one lease and a repeated GO is dropped.
  • Desync, stall and focus are recorded findings, not stops; only a finding unique to the build is RED, and signatures drop UUIDs, window ids and long hex.
  • Shrinking runs in separate short replay leases, and a --replay is shrunk only when --shrink-tries is given.
  • Verdicts are CLEAN/RED/NOT JUDGED/DIAGNOSTIC; CLEAN needs --base, positive --minutes, full fuzz time and no replay, and writes a .clean file.
  • Normal mode requires --whole-host with a key window; without one it reports "not run: known gap", never a product red.
  • Signals are blocked until the lease is inside its try and through clean-up, so a stop never kills the lease release; rounds from an unpinned harness checkout are "not run: harness unpinned".

R3b: New Column keeps every old column's exact width; Cmd-D and Split Down split inside a column without changing others.

Includes test_fuzz.py (33 tests), test_fuzz_sim.py (24 tests, 50 sim seeds clean), test_run_release.py (6 tests, including LateLease and DoubleStop), test_run_stale.py, test_run_stale_result.py and test_smoke_gate.py, all runnable without a fleet or app.