acpmux tests: test threads get the daemon's worker stack (base red: btw stack overflow)
Summary
Base red on feat-cmux-next 30806df8b37d (hq run 41212547003), still red at 87c359304617: two in-process hub tests abort with "thread ... has overflowed its stack / fatal runtime error: stack overflow, aborting" (they also fail when run alone):
fork_through::btw::a_side_session_opens_beside_a_running_claude_chat_and_never_disturbs_itfork_through::btw::a_codex_side_session_runs_sandboxed_read_only_and_the_adapter_is_refused
Culprit: merge d3f317b93cf, which brought in #28923's CPU-class start (debfed3dfdc, 73de34e2674). It made reach_or_start's cold-start future bigger: a retry loop around timeout(cold_start), plus starting_harness and enter_run_class awaits.
Bisect on the first-parent line, test btw:::
bfa1eb01afc(its first parent): pass, job 1181abd8fbd3f9273eb40775002bbd3cf98: pass, job 485a446dfb0fe2712bd44832d3f317b93cf: abort, job 29c57fcf1606333d55566cc2
No recursion, and the depth does not grow with the chat.
- The merge adds no recursive call. Its new awaits are
enter_run_class,harness_seen,with_live_harness,starting_harness,end_cold_startandabandon_cold_start, and none of them calls back into the start. - The stack the path needs is fixed, near 2 MiB. This was measured on the fleet, with one stack size per test process:
- The Claude test needed ≤ 2048 KiB before the merge and 2049-2304 KiB after it.
- The Codex test probe needed 2049-2304 KiB both before and after (jobs 9e0bf58944fa00ab78483395 and cedb1d7882a32555e3edd72a). The probe adds its own thread and wrapper, so its limits are a little high: the real Codex test passed on 2 MiB at
bfa1eb01afc(job 1181abd8fbd3f9273eb40775). Correction after the review.
- A side session after 1 finished turn and after 60 turns (2 KB each) needs the same stack: both abort at 1792 KiB and pass at 2048 KiB (job f9cfddd1616f602233009a20). A long chat does not make it deeper.
- On a 32 MiB thread both tests pass (job 6d0c1114b936cdd2022d6b5c).
Why only tests abort. The daemon never polls these futures on 2 MiB. Its runtime workers have WORKER_STACK_BYTES, which is 8 MiB (cli/entry.rs; it was set for exactly this reason after an adoption abort). A test that runs a hub in process polled them on libtest's 2 MiB thread, which is a stack the daemon never has. The daemon has about 3.5x margin on this path.
Fix. The hermetic test environment (tests/support/daemon_env.rs enter) now sets RUST_MIN_STACK to that same constant, which is now pub so the two places cannot drift apart. libtest's test threads and tokio's runtime threads in a test process then get the daemon's stack, so a test fails exactly where the daemon would: past 8 MiB.
- A host's own
RUST_MIN_STACKstill wins. acpmuxprocesses that a test starts get the hermeticvars, which do not carry this variable, so they keep their own worker stack.
The other red of that run, lane_reports::reads::lane_report_transcript_reply_read_consumes_its_ended_turn_not_state_only_pages, landed red with #28750 (8320cc3f047, job 75ca84467737b51d314649c7, left: 2 right: 1). It is already fixed on feat by 4532945f092 (cx-85ske2), which is after 30806df8b37d, so this PR does not touch it.
Owner: PL-acpmux (cmux-tui/crates/acpmux/).
Testing
| run | ref | result |
|---|---|---|
red: --arg test --arg btw:: | base 87c359304617 | both tests SIGABRT (stack overflow), 15 of 17 pass, job c9158689133ce4423b421535 |
| green: same | head 7084479b9434 | 17 of 17 pass, job a94abaf0ca7075cc0e71c3d3 |
clippy -D warnings | head | pass, job 8f3cc5fd3ae617470ff5f42c |
full workspace --arg test | 7084479b9434 (based before 4532945f092) | 13690 of 13691 pass. The only red is lane_reports::reads::lane_report_transcript_reply_read_consumes_its_ended_turn_not_state_only_pages, the base red fixed on feat by 4532945f092. No stack overflow. Job fc0bf0016c831be03c5f0f2d |
full workspace --arg test | merged head 4606fed59cb2 (feat merged in, plain merge, PR patch-id unchanged 20522f44fa71) | 13690 of 13691 pass. Both btw tests and the lane_reports test pass, and there is no stack overflow. The only red is cmux-tui-core local_resize_order::local_resize_bounds_the_drain_to_a_queued_snapshot (gate-notice FLAKY: failed in the full run, passed alone), in a crate this PR does not touch. Job a358e48a6949cb14f97b894d (hq run 41219579298) |
R9: no cap, queue limit or deadline is added. The test stack is the daemon's existing constant, not a new number. R10: no change on the main thread (only test processes change).
Observed / Not run
Observed: everything above, on fleet Linux overflow.
Not run: no Mac run of the two tests. The Linux numbers are the evidence.
Known limits (review notes, not blocking)
RUST_MIN_STACKapplies to every std thread in a test process that has no explicit stack size, not only libtest and tokio threads. A few daemon helper threads stay at 2 MiB in production but get 8 MiB in tests (for examplesrc/agent_host/wait.rs:103andsrc/harness_health/probe.rs:59). They are shallow I/O helpers. The comment indaemon_env.rsthat says a test fails exactly where the daemon would is therefore true for the runtime workers and libtest threads, not for those helpers.- The cold-start future grew by about 10% with #28923, and this PR does not make it smaller. With 8 MiB daemon workers that is acceptable, but tests now warn only at the daemon's limit, not earlier.
Changelog
none
Proof
No capture: a test-environment change. The daemon is unchanged.