test(acpmux): fuzz row g, kill -9 crash fuzz of durable answers

LA
5d ago
+1850-0

Summary

Fuzz row g (the fuzz catalog's acpmux row): a harness that sends random chat, draft, queue, outbox, watch and permission operations from three concurrent clients to a real acpmux daemon, with stand-in agents (tests/fake_agent.py). It kills that daemon with kill -9 at a seeded point, between requests or while other clients' requests are in flight. Then it starts the daemon again on the same ACPMUX_HOME and checks the result against a model of every answer the daemon gave.

What it checks:

  1. Answers the documents call durable survive the crash: _acpmux/draft_set ("the one durable shared draft"), accepted prompts (still queued or run), queue_remove removals (they do not come back), _acpmux/send_queue deliveries ("the ack ... across a restart too"), watch_add / watch_rm (durable-watches-v1), and created chats.
  2. No request waits more than 2 s for its answer while the daemon is up. It reports p50/p95/p99/max.
  3. The restarted daemon answers, lists no session twice and does not crash-loop.

A loss that no document promises (a rename, markUnread, session/set_mode, a permission decision record) prints a FUZZ-G NOTE line, not a failure. On a violation the test prints FUZZ-G VIOLATION seed=.. step=.. op=.. what=.., then replays the operations one at a time and shrinks them.

This is a measurement, not a CI test. Each test returns at once (FUZZ-G SKIP) unless the fleet step key contains fuzz-g or ACPMUX_FUZZ_SEED is set. The nextest ci profile keeps its output and gives it 300 s. No CI job, step or gate is added.

Testing

Fleet, Linux route (cmux-ci run --class light --os linux --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg fuzz_g_crash_durable), at this head 349c96f1a46 (feat after #29311 and #30120, plus this harness):

The review of 4268ad50ddc (comment 6554698585) found 4 oracle holes that could pass a real fault, and 3 weaker ones. All 7 are fixed. Each has a fuzz_g_oracle_* test that builds the model state the fault leaves and asserts the oracle reports it:

  1. A queue_remove answered removed: false no longer excuses the prompt; after each restart check, a removal whose answer the kill cut is settled by what the check found (landed: the prompt must never come back; did not land: it excuses nothing later).
  2. removed-back also reads the run log: a removed prompt that ran is a lost removal.
  3. A request cut by a kill counts its wait until the kill; a timed-out _acpmux/info counts its wait; either over 2 s is a stall.
  4. A run with no operation, no kill, or no checked prompt or draft fails (FUZZ-G NO-WORK).
  5. After each restart check, every register holds the value the check read, as one acknowledged write: an old unanswered write can no longer pass.
  6. An answer finds its write by a per-write number, not its Instant.
  7. Each daemon of one process gets its own home (a counter in the path), so plain cargo test cannot fake a g3 exit 75.

The self-tests call functions this change adds, so they cannot run against the old harness: they show the new oracle catches each case, not that the old one missed it (the review's reading of the old code shows that). The earlier green runs at 4268ad50ddc no longer count.

Findings this harness made (row g of the fuzz catalog):

  • g1 (stall): a prompt that came just behind another got no prompt_accepted until the first turn ended. Red test fuzz-red/acpmux-g1; fixed by #29175.
  • g2 (P0, lost durable answer): an older save of a chat's metadata could write over a newer acknowledged change, so kill -9 lost an acknowledged draft_set (and 8 other fields). Fixed by #29311 (one writer per chat, versioned commits, answer after the durable write) and #30120 (follow-ups).
  • g3 (candidate, not proven): one daemon restart in about 720 exited 75 (daemon_lock_held) right after the test killed and reaped its own daemon. Not seen again in 22 Linux runs. On a refused start the harness now prints FUZZ-G G3-DIAG lines (every process with daemon.lock open, its ps row, whether it was a child of the killed daemon, /proc/locks, the daemon's stderr) and keeps the home. fuzz_g3_seed_26401_restart_lock runs that seed alone.
  • g4 (candidate, under classification): on fleet Macs only, session/new sometimes takes 2.0-5.1 s (p50 about 2.3 s), mostly for the first chats after a restart. Linux shows none. An A/B with the stand-in at normal CPU class runs on one Mac (scratch branch). It does not change this harness.

Harness fixes found by its own runs:

  • markUnread: a prompt from the test's own connection shows the chat, which clears unread; the model now expects that.
  • queue_remove: a removal sent just before a kill may land with no answer; any prompt a queue_remove was sent for may be absent after the restart (seed 26369, job 8beafc5f7eea reported it as a lost prompt).

Earlier runs (before the g1 and g2 fixes): aa39e46a025b (head ca0aca4ce97e, 4 of 4 green) and ea2a7c69880b (head ff105ee3ea38, red on g1 only).

Observed / Not run

Observed: the FUZZ-G SUMMARY lines of the jobs above.
Not run: no local run (repo rule). On macOS this head is not green: candidate g4's stalls break the "no wait over 2 s" check there. The shrinker ran on fleet violations (g2, the queue_remove gap); it gives no short list for timing events.

Changelog

none

Proof

No capture: tests only.