PR #24290 - manaflow-ai/cmux

Refs manaflow-ai/cmuxterm-hq#2995
Leo-ask: ask-42afb6109a
Leo-ask: ask-427cc7d8f5 (Leo: "the whole app froze btw i got codex to put some issues down in cmuxterm-hq")

Why

Leo: "the whole app froze btw i got codex to put some issues down in cmuxterm-hq", then "gray is an m5 pro with 48gb of ram... just limit it nicer", "cosmic gray no it should be more cpu aware" and "i mean cmux next itself should be more cpu and ram aware... with stat or whatenot. maybe even acpmux itself.. just warn for now instead of prenveting stuff".
claude-sr2 (relaying): "Backpressure should also cap per-host concurrent turns and background process counts (a host-level budget)". Leo's later warn-only order above supersedes it: this PR warns and holds nothing.

The incident (cmuxterm-hq#2995): cosmic-gray (15 CPUs, 48 GB), 23:06 to 23:20 PDT on 2026-10-21:

15-minute load of 146;
34.7 of 36.8 GB swap in use;
about 355 MB/s of disk I/O and 0 to 2% idle CPU;
39 agent turns (9 local, 30 on peers) and 1,298 live background processes.

acpmux had no view of the host's load.

What changes (this PR: the budget itself; the daemon wiring is the next PR)

This is split to stay under ~800 lines:

This PR adds host_pressure.rs and host_pressure_probe.rs, with no runtime caller yet.
The next PR (branch app-eff/acpmux-host-pressure-status, opened once this one lands) wires it in: daemon status --json gains host_pressure, a turn's acceptance carries it while over budget, and the CLI prints the warning line.

evaluate(host, sample, swap_mb_per_min, budget) -> Verdict is the one function that decides. Nothing in the budget is hand-set: it all comes from the host. The host is over budget while any of these holds:

Load: the 1-minute load is at least load1_per_core × cores (default 1.0).
- macOS counts performance cores: every perf level not named Efficiency (hw.perflevelN.name / physicalcpu), else hw.physicalcpu. An M5 Pro names its levels Super (5) and Performance (10), so cosmic-gray counts 15. An M5 (air-blue) counts its 4 Super cores. M1 to M4 count their P-cores.
- Linux counts the CPUs this process may use (nproc).
Memory pressure is above pressure (default normal).
- macOS: kern.memorystatus_vm_pressure_level (1 normal, 2 warn, 4 critical).
- Linux: /proc/pressure/memory some avg10 ≥ psi_some_avg10 (10).
Available RAM is under min_available_ram_fraction (0.10).
- macOS: kern.memorystatus_level.
- Linux: MemAvailable / MemTotal.
Swap-outs grow faster than swap_growth_mb_per_min (64).
- macOS: Mach vm_statistics64.swapouts (vm_stat's Swapouts).
- Linux: /proc/vmstat pswpout.

Why these numbers:

A load equal to the performance-core count means the fast cores are saturated. The incident's 146 was about 10× cosmic-gray's 15.
"Normal" memory pressure is the macOS level at which the desktop stays smooth.
Growing swap is what turned the overload into beachballs. A large swap that stays flat is harmless.
10% available RAM matches lane-place's RAM reserve.

Keys and overrides:

The keys are deputy-2's host-budget contract: the same as leo-workspace machines/host-budget.json / tools/host_budget.py, so lane-place, the app and acpmux agree.
The defaults are built in.
$ACPMUX_HOST_BUDGET, or else ~/.config/cmux/host-budget.json, overrides them: formula first, then hosts.<short host name>.
Only a regular file of at most 64 KiB is read. A FIFO, a device, a big file, a bad value or a bad file keeps the defaults.
enforce is reported, not acted on.

The verdict has the contract's fields: host, cores, cores_kind, load1, load_limit, pressure, available_ram_fraction, swap_mb_per_min, over[], ok, warning, enforce, formula. Its warning line looks like this:
host-budget: cosmic-gray over its CPU/RAM budget: load1 23.0 >= 15 (15 P-cores); swap +120 MB/min

Nothing is refused, queued or held. No process is killed or stopped.

Timers

This PR adds none, and nothing calls it at runtime yet.

In the next PR, the daemon samples only when a request asks (daemon status, a turn's acceptance):

at most once per sample_seconds (3);
on a blocking thread, one sample at a time;
status waits at most 250 ms, then answers with the last verdict;
a turn never waits: it repeats a verdict at most a minute old.

Nothing runs between requests, so there is no stop condition or cleanup to name.

Review

Codex xhigh found a BLOCK at 249515597b1: the override file was read synchronously under the meter lock, inside a turn's acceptance.
- The fix to the meter moved to the next PR with the meter.
- The regular-file and size check stays here.
Codex's non-blocking notes are addressed here: fixtures for the /proc parsers, and a test that the verdict's JSON keys match the contract.

Interface for the app (air-blue/app-eff-pressure-ui)

With the next PR, the app reads host_pressure from _acpmux/status:
- warning is the text it shows;
- ok, over, load1/load_limit, pressure and swap_mb_per_min drive the indicator;
- top fills the popover.
No pause/resume or stop-background-lanes operation: Leo's order is warn only. Stopping a lane already exists as acpmux session stop NAME (the session stays resumable).

Not in this PR (follow-ups)

The daemon wiring (the next PR, above).
enforce: true hold/queue ("waiting for CPU") behind that key, which defaults off.
Starting the background git and lint that acpmux spawns at background QoS.
One fsmonitor daemon per worktree (cmuxterm-hq#2995 item 5).

Tests

host_pressure_tests.rs uses the incident's numbers and a calm-host sample. It covers:

the core fallback (performance, then physical, then logical) and cores: logical;
the perf-level fold for cosmic-gray (M5 Pro, 15), air-blue (M5, 4), cmux16 (M6: Super 2 + Performance 4 + Efficiency 6 = 6) and M1 to M4;
each limit tripping alone, with the boundary at exactly the limit;
PSI on Linux;
the file overriding the defaults and the host entry overriding the file;
bad values and bad files keeping the defaults;
the override file being refused when it is a FIFO, a directory, too big or missing;
the /proc parsers;
the verdict's JSON keys.

Before / after (same readings, cosmic-gray)

One harness for both columns: the same cosmic-gray readings (uptime, sysctl, vm_stat; nothing built or run there), judged by base and by this budget.

reading	before (base)	after (this PR's `evaluate`)
2026-10-21 23:10 PDT, the incident: load1 146.3, pressure warn, swap growing	acpmux has no view of the host: no field, no warning	over: `load1 146.3 >= 15 (15 P-cores); memory pressure warn; swap +N MB/min` (the incident test)
2026-10-22 00:54 PDT: load 17.77 / 21.65 / 23.41, pressure 1, free 56%, swap 26.0/27.6 GB	nothing	over: `load1 17.8 >= 15 (15 P-cores)`
2026-10-22 20:13 to 20:17 PDT: load1 23.40, then 29.97; pressure 1 (normal); free 46% then 49%; swap 31.9/32.8 GB used; Swapouts 268,731,013 pages at both reads (0 MB/min)	nothing	over: `load1 30.0 >= 15 (15 P-cores)`. Memory pressure, RAM and swap growth are within budget: the swap is full but flat, which this budget does not count.

With the old perflevel0-only count (5 cores), every one of those readings, and any load over 5, would have warned. With 15 performance cores, the warning fires only when the 1-minute load passes the core count.

The warning in the daemon is the next PR; it carries this same verdict.

CI

At 59a3e9e4166 (the lander's base update over af973095c6c; the four files are byte-identical), the only reds are base reds:

Guard status and release-notary are on the rulebook's don't-block list.
cmux-next checks has three red items:
- "No new SwiftUI" and "Checks job structure" are red on base run 39892307066 too.
- "No fleet host names in gallery fixtures" is red on base run 39750405912; this PR touches no gallery fixture.

Other results:

Fleet cmux-ci compile-check --ref 59a3e9e4166537f8ce273222e772a63caa96612a --app: passed (job d9f68bccdeefc6d1ee1e83cb, 65 s).
Hosted verify-cmux-tui-hosted.sh --filter host_pressure at af973095c6c: Linux Rust tests, lint and MSRV passed. The hosted macOS job was cancelled, so it is being re-run at this head.

Observed / Not run

Ran locally (air-blue): rustfmt and scripts/cmux-next/prepush-gate.py.
Not run locally: cargo build and tests. Cargo never runs on Leo's Macs; the hosted runners ran them (above).

Changelog

none (the budget has no caller until the next PR)

Agent session: Air-Blue

🤖 Generated with Claude Code