acpmux: stdio MCP servers run on two tokio workers, not one per core
Why
A fleet host's acpmuxsessions.slice hit its TasksMax=16384, with 15,927 to 16,348 tasks across 105 session scopes, so forks failed at random. The biggest scopes held 280 to 600 tasks.
The per-process breakdown came from two other 32-core lane hosts with the same setup, whose biggest scopes held 377 to 484 tasks:
- Codex itself held about 46 threads.
- Most of the rest came from acpmux's own stdio MCP servers:
acpmux-mcp,subsession-mcpandforum-mcp, at 34 threads each.
Codex starts one set of these servers for every thread it loads, so each subagent gets its own set, and it keeps that set until the app-server exits. Each server ran a tokio worker per core. One set therefore cost about 100 tasks on a 32-core machine, and a session with 5 subagents spent about 600.
Leo's standing order is: "Build systems, not chores ... A manual step or a new hard-coded constant is not a fix." The slice cap was raised by hand. This PR removes most of the load that filled it. #27555 makes the cap itself typed and machine-scaled.
What
The six stdio MCP servers acpmux registers for an agent session (acpmux-mcp, subsession-mcp, forum-mcp, offer-mcp, secret-mcp, cua-mcp) now run on a 2-worker multi-thread runtime. Every other command keeps tokio's default runtime. Each server answers one client on one pipe, and it stays multi-thread so block_in_place keeps working.
The codex side is tracked upstream:
- openai/codex#38247: completed v2 subagents keep their full stdio MCP runtimes, reported on 0.147 through 0.164.
- openai/codex#37870: the same on the CLI.
- openai/codex#38353: a proposed MCP broker for subagents.
This PR makes each retained set cheap; it doesn't stop codex from keeping them.
Performance
Measured on the same cmux-ci 32 vCPU Linux runner type: start the server with stdin open, wait 4 s, then count /proc/PID/task.
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
threads per forum-mcp (32 cores) | 34 | 4 | -30 (-88%) | before https://github.com/manaflow-ai/cmux/actions/runs/40955691416, after https://github.com/manaflow-ai/cmux/actions/runs/40955685963 |
threads per acpmux-mcp | 34 | 4 | -30 | same |
threads per subsession-mcp | 34 | 4 | -30 | same |
| tasks per MCP set (3 servers) | ~102 | ~12 | -90 | derived |
On the incident host, about 100 MCP sets were running at once. Saving about 90 tasks per set takes roughly 9,000 tasks off the slice. This PR adds no new timers, observers or loops.
Red, then green (cmux-ci, 32 vCPU Linux)
- Red, tests only at fecd483b884: https://github.com/manaflow-ai/cmux/actions/runs/40954005788. It fails to compile because
runtimeandMCP_WORKERSdon't exist yet. - Green at a024336b5a1: https://github.com/manaflow-ai/cmux/actions/runs/40955685963. It ran:
cargo test -p acpmux --lib -- cli::entry: 3 passed.cargo test -p acpmux --lib -- mcp: 14 passed.cargo clippy -p acpmux --all-targets -D warnings: clean.
Risk
Low.
- An MCP server that does heavy parallel work would now run on 2 workers. These servers forward requests to the daemon over a socket and are I/O bound.
block_in_placeneeds a multi-thread runtime, and it still has one.- Command detection matches the first non-global argument exactly, so only those six hidden subcommands change.
Agent session: agent session
š¤ Generated with Claude Code