runner-hook: a nightly signing drain outlasts the running jobs, and a handed job never fails for a fleet waiter

Merged
TE
7d ago
+128-42

Why

Legacy nightly signing on owned minis was refused on every attempt on cmux12s and cmux13s since about 15:59Z 2026-10-30. That held even after cmux#28642 listed hook f612f5a8. Example: NEXT nightly run 41161589041 (a7ebd2a9), sign-nightly-identity-legacy on cmux12s-mac-mini-glaeda-2, was refused with capacity: 6 of 14 units free ... needs 14 after wait_s 600.3.

Rulebook: "a waiting nightly, promote or nightly sign/verify job is admitted on a mini ahead of every PR, DEV and side job ... and no new PR job starts on a mini where a nightly job waits."

Root cause (from glaeda-cmux-jobs.jsonl on cmux12s and cmux13s, 2026-10-29 to 10-31)

  1. The drain was too short. Its hold was not the problem. There were 23 capacity refusals. Each one waited the full 600 s, and the hook admitted no job at all on that mini during any of those waits, so the hold worked. Each time, a job that started before the drain still held units. On 21 of the 23 it was a cmux-next swift-test on the gui runner (p50 1962 s, p90 2368 s on those minis), and it ended 1 to 43 min after the refusal. The 600 s bound assumed a ~7 min compile. The longest job those minis ran in the window took 3203 s (n 1035, p99 2410 s).
  2. The "gateway-0 token is taken" refusals are the same timeout. The last note was a gateway token held by a fleet test-compile client that started before the drain.
  3. The "a fleet build is waiting for the host" refusals were immediate (cmux13s 03:05Z: three signers in 30 s) or came mid-drain. A nightly job ahead of a fleet waiter that could not take its capacity on the first look was refused outright (refused() -> WAITER_REFUSAL). It did not keep waiting.

The same refusal is what failed cmux run 41150189497's swift-test (cmux#28544) red on cmux12s: a fleet build is waiting for the host (pid 15148), wait_s 264, no test ran.

What

  • SIGNING_DRAIN_WAIT_S 600 -> 3600 (SIGNING_DRAIN_OUTLASTS_RUNNING_JOBS). Only a nightly-tier signing job on the nightly slot gets it; other signing jobs keep EXCLUSIVE_CAPACITY_WAIT_S (300). cmux's owned-pool rescue never judges this job: owned_pool_rescue.nightly_watched watches only build-nightly-app and trusted-pool jobs of a nightly run, and the slot's label is glaeda-nightly-std-xcode-*. So the longer wait cancels nothing. It fits in the job's timeout-minutes: 160.
  • HANDED_JOBS_NEVER_FAIL_FOR_A_WAITER: in job-started (new_job), a waiting fleet build no longer refuses a job GitHub already handed the runner. The job takes its units like any capacity wait, and the build gets the host when the job ends. The listener gates already stop the idle runners while a fleet build waits, so no further job arrives. Neither side waits on the other: the drain waits for Runner.Worker to exit, and the job waits only for units held by running jobs, never for the build. A ci-step lease (step_admit) is still refused, and the build worker leases it elsewhere. Nightly passes (NIGHTLY_WAITER_PASSES) are still counted, so the nightly slot's gate still yields to a waiter once its passes are spent. The "a nightly job goes first" line is logged once per waiter, not on every retry.
  • docs/CMUX_MINI_RUNNER.md updated for both.

This adds no new admission or priority rule. It makes the existing nightly-first drain finish, and it stops handed jobs from failing.

Tests (fail on main)

  • test_a_nightly_signing_job_on_the_nightly_slot_drains_the_mini: the bound is at least 3203 s and leaves the job's 160 min room (old: 600).
  • test_a_draining_signing_job_keeps_draining_when_a_fleet_build_starts_waiting: a signer with a fleet waiter and a busy unit waits, then takes the whole mini (old: refused at once).
  • test_a_handed_job_is_admitted_while_a_fleet_drain_waits_and_never_fails_red, test_a_handed_job_is_admitted_while_a_fleet_build_waits_for_the_lock, test_nightly_job_is_admitted_while_a_fleet_build_waits: handed jobs are admitted, and the build gets the host after they end (old: refused: a fleet build is waiting for the host).
  • Unchanged and passing: ci-step refusal (test_ci_step_fits_nothing_while_a_fleet_build_waits), the nightly-pass and claim-gate tests, and the nightly slot gate tests.

I ran a local -k selection of 50 tests. One failed, test_capacity_wait_continues_when_the_refusal_turns_to_units, and it fails the same way on unmodified main on this host (timing). Verify runs the full shards.

Rollout follow-up

The new hook's sha256 has to be added to cmux scripts/ci/glaeda-signing-hooks.txt (as #28642 did) before an owned-Mac claim gate accepts it. The cmux comments that say "600 s" (signing_runner.py, nightly.yml) are stale after this change.

Agent session: agent session

šŸ¤– Generated with Claude Code