acpmux: a prompt right behind another is accepted queued at once (fuzz g1)

Merged
LA
6d ago
+441-34

Summary

Fuzz finding acpmux g1 (fuzz row g, seed 26370; STATUS 1052): a prompt that comes just behind another prompt in the same chat got no _acpmux/prompt_accepted until the first prompt's whole turn ended. With a long turn, the sender got no answer at all. In the app this reads as a lost message: the composer keeps the text, and Enter and Send do nothing. No durable answer was lost.

Cause: a prompt takes turn_lock in queue_slot, but its turn shows in session.turn only later, in start_turn, after the awaits of check_dispatch, family_dispatch_check and hold_retry_start. queue_slot read only the turn and the queue count, and its check and its count increment were two steps. So a second prompt in that window (or one that came at the same time) did not queue. It waited on turn_lock.lock().await for the first prompt's whole turn, with no acceptance.

Fix (acpmux, hub/queue.rs, hub/subsession_runs.rs, hub/session.rs):

  • The prompt that takes the lock for its turn marks it (Session::turn_claimed, set by ClaimedTurn with the lock, cleared before the lock is released). queue_slot counts that mark as a turn.
  • The check whether a prompt must wait and its increment of the queued count are one step under a new leaf lock, Session::admit (no await under it; order admit, queue, turn; only queue_slot takes it). Two prompts that come together never both find the chat free.
  • So the second prompt is journaled (its queued record) and accepted at once with queued: true, state: "queued", position: n. Its turn's records follow when its turn starts.
  • position is now the prompt's real place in the queue (insert index + 1, so 1 = next), as an inbox digest already reports its own. Before, it was the queued count, which a prompt about to start its turn is still in. The journal's queued record carries the same value.
  • An automatic continuation (after_turn) never queues, so it still waits for the lock and check_cut decides after, as before. A no_queue subagent retry still gets retry.busy.
  • The acceptance says what happened, additively: state is queued, started or steered (docs/acpmux-schema.json). An older client reads queued and steer as before; the webview (direct.ts), the Swift pane, cmux-tui-core and the CLI read only promptId and queued. A duplicate or inbox acceptance sends no state.

No client timer and no client-side guess: the daemon answers when the prompt is in its journal. The fuzzer's 2 s oracle is unchanged.

Behavior changes (a prompt sent while the earlier prompt is still starting is now a queued prompt, so it acts as any other queued prompt does):

  • If the earlier turn fails after its prompt reached the agent, the queue pause holds this prompt. Before, it ran.
  • A Stop pressed while the earlier prompt starts refuses only that prompt; the queued one runs next. Before, both were refused.
  • An agent message that comes while an agent-message turn starts can join that turn's digest. Before, it ran as its own turn (first_work_tests now expects turn_count 1).

Clients: the app's agent pane already draws a queued prompt as a queue row from the queued record (no number by design, Leo 2026-10-21), so it needs no change: the fix makes the record and the acceptance exist in this race. The CLI (acpmux send --no-wait) prints queued as before.

Testing

All runs are on the Linux fleet (cmux-ci run --class light --script scripts/ci/cmux-tui-rust-check.sh). Nothing was built on a developer machine. PR head: 31771d801ee41836bf5efc8cf9edcdbc6b008ba1. Each job names its exact commit. Two commits are scratch trees, never merged:

  • c8988bc0992c96c8974a02f9dccb15ca59ed0691 (fuzz-red/acpmux-g1-unit) = the red test commit 12e2f3f1ef6 + prompt_accept_tests.rs only.
  • a585103a3733c5eb1760c32a3ef0ec964df73721 (fuzz-red/acpmux-g1-fix-31771d80) = the PR head 31771d801ee + the 2 row-g harness commits of #29103 (1673bda5a4f, a585103a373; they add only tests/fuzz_crash_durable.rs and .config/nextest.toml).
WhatJobExact commitResult
Red: test fuzz_g153abab535b78cfd4af4e684812e2f3f1ef61bbcd50598ad7f8fe29c12bd8492f (test commit)FAIL fuzz_g1_a_prompt_right_behind_another_is_accepted_within_two_seconds, FUZZ-G1 silent=14 of 20; control passes
Green: test fuzz_g126c66c5b91f5e5257a90f8d231771d801ee41836bf5efc8cf9edcdbc6b008ba1 (head)3 of 3 pass
Green: test fuzz_g1_ (fuzz owner)c74bf1f299becad8bd95ece131771d801ee41836bf5efc8cf9edcdbc6b008ba1 (head)3 of 3 pass
Red: test prompt_accept_tests310423536f1472badb2e7649c8988bc0992c96c8974a02f9dccb15ca59ed0691 (scratch: base + test)FAIL prompt_accept_tests.rs:61 "the second prompt is accepted before the first turn ends: Elapsed"
Green: test prompt_accept_tests7c109d38cdaa7c02c1ffdb7831771d801ee41836bf5efc8cf9edcdbc6b008ba1 (head)pass
clippy0ec29d124c7f9cfb88f3198631771d801ee41836bf5efc8cf9edcdbc6b008ba1 (head)green
Fuzz row g, run 1 (test fuzz_g_crash_durable)dfb5bc06bd757ea5628141ffa585103a3733c5eb1760c32a3ef0ec964df73721 (scratch: head + harness)4 of 4 seeds, 5,955 ops, 144 kills, 0 violations (0 stalls, 0 durable losses), p99 86-93 ms, max 173 ms (feat: max 22,105 ms, 3 stalls)
Fuzz row g, repeat 4 of 5f1a586e8613b75c0f74e863ea585103a3733c5eb1760c32a3ef0ec964df73721 (scratch: head + harness)4 tests pass
Full test602fe20ef88ac341935d8da731771d801ee41836bf5efc8cf9edcdbc6b008ba1 (head)13,532 of 13,543 pass, 11 fail
Full test (base)60e35145dcccb1b3dd7382e2400b94dba51f5bf321cb257e165cb1b652404dce (base)13 fail

10 of the 11 head fails also fail at the base (cmux-tui::setup x6, cli_audit, peer_restart, startup_load, subsessions_safety). daemon_list_guard::clients_asking_for_the_whole_list_over_and_over_cost_the_daemon_a_few_percent failed once at the head and passed on the solo rerun in the same job: a CPU-budget flake, not in the queue path.

Earlier head, superseded (for the record only): 31eeaa9f154 green fuzz_g1 job ce7a383f04fe5a957d7335cc (3 of 3).

Observed / Not run

Observed: the fleet runs above, red and green, each on its own assertion.

Not run: a live app check in a tagged build (the pane code is unchanged). The admit step has no deterministic red of its own (the unit test drives the claim path); the 20-trial socket test and fuzz row g cover it. The fuzz owner adds the remaining row-g repeats (5 of 5) as comments, each on a585103a373 (head + harness).

Changelog

Fixed: A prompt sent just behind another one in an agent chat shows as queued at once, not after the first turn ends

Proof

No capture: daemon change only; the pane's queue row is unchanged.