scripts(cmux-next): Austin's day, the P-AUSTIN load scenario (fleet bench)

Merged
LA
6d ago
+2952-0

Summary

Austin's nightly-next stopped working on 2026-10-29 under about 80 terminal agents (#27698, #27702, #27693): the daemon's 64 connections were full, the app's own requests were refused, workspaces would not close, and swap was nearly full. This PR adds scripts/cmux-next/austins-day/, a fleet bench that puts the same load on an exclusive capture mini against a tagged build of one commit and measures it. The P-AUSTIN fixes are measured on it and never tuned against it. Its owner, the P-AUSTIN harness owner (agent pa-d), runs each fix's A/B and posts the result on that PR. It is not a CI job or a gate. The run commands are in the README.

  • Load:
    • 80 claude runs in real cmux terminals, through the cmux shim, cmux-tui agent claude-wrapper and its hook settings. A fake Claude runs those hooks on every tool call and stop, as Claude Code does (sync hooks are waited for, async ones are not, and the time-out applies). It also calls the cmux CLI and draws to its terminal while it thinks.
    • A --filter=blob:none partial clone with 19,110 promisor packs, built once per host.
    • A wave of 25 acpmux chats every 60 s.
    • A bounded memory hog that pushes swap up.
    • A workspace close through the app's closeWorkspace action every 30 s.
  • Output: one JSON plus a text summary with p50/p95/p99/max and counts for:
    • app requests (the debug.daemon_requests probe, #28295) and the "no reply within" lines in the log;
    • refused connections, as a rate per second and as a share of connects;
    • who holds the daemon's connection pool;
    • lock wait and hold;
    • workspace close time;
    • acpmux chat-open failures;
    • hook latency and CPU;
    • memory, fds, load average and swap;
    • stalls over 100 ms, split into CODE (a red, named by its top cmux symbol and its OWNERS.md owner) and WAIT.
  • A/B rules:
    • The base is the PR's parent.
    • The arms alternate in ABBA order, 5 rounds.
    • Each metric gets a paired t-interval (95 %) and a run-position estimate.
    • A result is invalid when a base did not reproduce Austin's failures (overall or for that metric), when the head edits the harness, when the base is not an ancestor of the head, or when the harness itself failed in a run.
    • An A/A self-test (--aa) checks the harness; its record is aa-failures.json.

Testing

  • Selftest: python3 scripts/cmux-next/austins-day/selftest.py passed, 16 tests, on cmux-orchestrator-1. It covers the fake Claude's hooks, a refused acpmux connect running through the driver's own code, the pack fixture, the summary math, the validity rules, the t-interval verdicts, the A/A mode and older runs.
  • A/A, same build in both arms (84c705ad, cmux21, 5 rounds): in ABBA order, 28 of 28 metrics pass (31 of 31 with the later acpmux rows). In the order run, acpmux RSS growth followed the run position. Details: https://github.com/manaflow-ai/cmux/pull/28077#issuecomment-6525676764
  • Fleet runs: run 1 on the feat tip 146a97390cf, the #28159 A/B and its correction, and the running #28041 A/B. Their results are in those PRs' comments.
  • Review: an independent review subagent reviewed the head in rounds. The final verdict is SHIP at 035db354e6a (receipt https://github.com/manaflow-ai/cmux/pull/28077#issuecomment-6526330455; the earlier rounds are in the comments above).

Observed / Not run

  • Observed: the selftest, the A/A, run 1, and the #28159 A/B and its rerun.
  • Not run:
    • An A/A of this final head. Its measuring code changed after the A/A (close timing, chat record, fixture reuse), so its A/Bs count only after its own A/A, which is next.
    • The 60-minute arm. It is blocked: a capture-host lease lasts at most 10 minutes, and cmux-ci run takes only allowlisted scripts.
  • Shape gaps against Austin's day: load average about 250 (his 150-252), swap about 10 GB (his 17), lock wait p99 655 ms (his 17 s). acpmux restarts were not seen, but acpmux fails most chat opens from the second wave on (owner pa-b10).

Changelog

none

Proof

No capture: test tooling only; nothing in the app changes.

Performance

No perf numbers: this PR adds a measurement harness and changes no product code, so it has no before/after of its own. The harness measures latency in other PRs; their A/B results are posted on those PRs.