runner-hook: a lock holder keeps its locks while its owner's start token is unreadable

Merged
TE
6d ago
+49-2

Why

cmux-next swift-test runs are being cut at their 1500 s limit because lent-worker ci-steps run outside the capacity ledger. A ci-step's lock holder (the detached process hold() leaves behind to keep the step's units and tokens) exits about 4 minutes after admit while the step keeps running, so its units go back to the ledger and the step runs uncharged beside PR jobs.

Cause: the holder's loop was while process_alive(watch_pid, identity), which runs ps -o lstart= with a 2 s timeout and reads a timeout or failure as "owner gone". At the load these minis see (144 to 228), ps times out. JobOwner.gone() already treats an unreadable start token as unknown, not gone; the holder loop did not use it.

Evidence (2026-10-30):

  • At 21:26Z, 15 of 16 running ci-steps fleet-wide had no holder.
  • On cmux11s (21:27Z, load 228) a lent-worker ci-step's holder exited four minutes into a two-hour suite step while the build worker it watched lived on.
  • swift-test p50 went from 600-800 s to 1800-2100 s since about 10-29 09Z; 306 of 443 swift-test runs in 24 h went to 1500 s or more.

What

  • hold()'s holder loop becomes while not owner.gone() and ...: an unreadable start token keeps the locks; a free pid or a pid that now names another process still ends the holder (proven gone).
  • HOLDER_MAX_S still bounds the holder.

Test

  • New test_a_holder_keeps_its_locks_while_the_owners_start_token_is_unreadable: patches process_identity to return None for the owner, checks the holder and its lock survive 20 polls, then kills the owner and checks the holder exits and the lock is free. Fails on main, passes with the fix.
  • The 54 holder/owner/hold tests of scripts/test-glaeda-cmux-runner.py pass on the merged head (main 63ace48b merged in).

Not changed: how cmuxterm-hq charges hook step classes (a separate policy question).

Agent session: agent session

šŸ¤– Generated with Claude Code