acpmux: typed config for session task caps; slice cap scales with threads-max

Merged
TE
7d ago
+306-34

Why

A fleet host's acpmuxsessions.slice ran into its fixed TasksMax=16384: 15,927 to 16,348 tasks in use, so forks failed now and then across all 105 session scopes. Memory had 44 GiB free and the kernel's threads-max was 514402, so the slice cap was the only limit hit. The largest scopes were codex app-server sessions with 280 to 600 threads each, well under the per-session 4096, so the shared slice was what bound. It was relieved by hand by raising the slice cap through an env drop-in on the daemon unit. That is a manual step, and Leo's standing order is: "Build systems, not chores ... A manual step or a new hard-coded constant is not a fix."

What

  1. Typed config keys, next to the existing session.memoryMb and session.swapMb in config.json:
    • session.tasksMax is each session scope's cap.
    • session.sessionsTasksMax is the shared slice's cap.
    • Both can be set with acpmux config --set session.tasks_max=N or session.sessions_tasks_max=N. 0 or infinity means no cap, and default clears the key.
    • Order: typed config, then env (the profile's, then the daemon's), then the computed default. ACPMUX_SESSION_TASKS_MAX and ACPMUX_SESSIONS_TASKS_MAX keep working.
  2. One reader for the slice and every scope. limit_sessions_slice, the session scope's properties and the agent-host scope (host_tasks_max) all go through knob_reader with the daemon config's limits. Once the config names a cap, profiles whose env disagrees no longer take turns rewriting the shared slice at each scope start (slice_changes). Before this, the host scope read only the daemon env.
  3. A slice default that scales with the machine: min(threads-max / 4, 262144), never below 16384. It reads /proc/sys/kernel/threads-max and falls back to 16384 if the file can't be read.
machine (threads-max is about RAM / 128 KiB)threads-maxslice default
8 GiB~64k16384 (floor)
16 GiB~128k~32k
the incident host, 64 GiB514402128600
128 GiB and up1M+262144 (ceiling)

The per-session default stays at 4096. The incident's largest session held about 600 tasks, so the per-session cap was never the one that bound. Every scope still gets an explicit TasksMax, never systemd's DefaultTasksMax:

  • All four property sets that prefix tries emit TasksMax=N or TasksMax=infinity.
  • The agent-host scope sets TasksMax through busctl.

Red, then green (cmux-ci, 32 vCPU Linux)

New tests:

  • typed config wins over env;
  • env wins over the computed default;
  • the threads-max scaling, its floor and ceiling, and an unreadable or junk value;
  • 0 and infinity behave as before, through both config and env;
  • the slice and the scope read one source, so two profiles with different env ask for the same slice value;
  • the CLI setter, and a camelCase JSON round trip;
  • a hub config reload carries the new keys.

Risk

Low to medium.

  • On large machines the slice's default cap rises from 16384 to as much as 262144. A fork storm spread across many sessions can therefore take more of the machine before the slice stops it. Each session's 4096 cap still holds, and a quarter of threads-max leaves three quarters for everything else.
  • For the two task caps, the typed config now wins over a profile's env. The memory knobs keep their order, where the profile's env wins. A profile that sets ACPMUX_SESSION_TASKS_MAX loses to session.tasksMax once that key is set.
  • A host that pins ACPMUX_SESSIONS_TASKS_MAX in its daemon env keeps that value, because env still wins over the computed default.

Agent session: agent session

šŸ¤– Generated with Claude Code