acpmux: an idle Codex session behind npm's launcher hibernates

Merged
TE
7d ago
+170-24

Why

A fleet host's acpmuxsessions.slice ran out of tasks (about 16,000 of 16,384), and forks failed across every session. #27555 raised the slice cap and #27606 cut the MCP servers' threads. Both have merged. The remaining cause was that idle Codex sessions never hibernated:

  • 59 Codex sessions with status ready, with no turn, nothing queued and nothing attached, each held about 150 to 180 tasks until the daemon stopped.
  • The daemon log showed the same refusal for each of them every 10 minutes: idle harness stays up: a process that is not one of its MCP servers runs under it ... comm=codex.

The process that refused was Codex itself. The npm package starts node ~/.local/bin/codex ARGS, a launcher that runs the native .../vendor/<triple>/bin/codex ARGS with the same args and relays stdio. The idle census treats the session's harness as the launcher. It found the native child, saw that it was not an MCP server, and kept the harness up. The real MCP servers and Codex's codex-code-mode-host helper are children of that native binary, not of the launcher, so they were never matched either.

What

hub/idle_census.rs classify now accepts two more kinds of child. Everything else keeps the harness up, as before.

  1. The launcher's relayed program. The leader's own command line must be an interpreter running a script by absolute path (node /abs/codex ARGS). A direct child counts as the harness when it runs an absolute-path program with the script's file name and exactly the same args. That child is then treated as the harness: its children may be MCP servers, and its helpers are its own.
  2. A helper of a relayed program. This is a direct child running an absolute-path program in the same folder, whose name is the relayed program's name followed by a dash (.../bin/codex-code-mode-host beside .../bin/codex). Only the helper itself is taken. Anything running under it is still judged on its own, so a sandboxed command the helper started keeps the harness up.

agent_host/tree.rs under now also returns the leader's command line from the same ps read (leader_args_in). It does no extra process scan.

What still keeps a harness up (all covered by tests):

  • anything else under the relayed harness (python3 -m http.server);
  • anything under its helper;
  • a same-named child with different args (codex exec ...);
  • a helper outside the relayed program's folder;
  • a program whose name does not start with the harness's name plus a dash;
  • a leader that is not a launcher (the native binary started directly);
  • a launcher started by bare name rather than a path.

Red, then green (cmux-ci, 32 vCPU Linux)

cargo test -p acpmux --lib -- hub::idle_census agent_host::tree

Observed / Not run

  • Observed on the incident host, read-only: the process trees and task counts of the scopes, and the daemon's refusals (comm=codex, 17 in the last hour after a reboot). The tree shape matches the new test fixture.
  • Not run yet: a live hibernation of an idle Codex session with this build. That follows the normal daemon update.

Risk

Normal. This changes only which processes the idle census accepts.

  • The worst wrong case is ending a harness while an absolute-path program, with the launcher's script name and the same args, runs as its direct child. That is the shape of a launcher's own relay.
  • Hibernation keeps the record and the agent session id, and the next prompt resumes the session. Work the agent left running still keeps the harness up.
  • This adds no new timers, observers or loops.

Overlaps #27558 (the census reads argv exactly, so a path with a space matches). That PR changes the same classify, but along a different axis. Whichever lands second rebases.

Changelog

Fixed: idle Codex sessions now hibernate, freeing their processes and threads, instead of staying up until the daemon stops.

šŸ¤– Generated with Claude Code

Agent session: agent session