ci(owned-pool rescue): re-run a job glaeda's hook refused after a long capacity wait

TE
7d ago
+74-2

The ask: "A capacity wait became a red that the PR owner chases." Fix it so a job refused at the hook never ends red.

What happened

Job 122357989139 ("cmux-next app compile (build fleet)", run 40551896747) spent 766 s in glaeda's capacity wait inside "Set up runner" ("1 of 10 units free, app-compile-fleet (light) needs 2"). A fleet build then started waiting for the host, and the hook refused: refused: a fleet build is waiting for the host. The owned-pool rescue counts a failed setup step as a refusal only within REFUSAL_SECONDS (360 s), so it read this as the code's failure and left it red.

Change

refused() now treats a failed "Set up runner" step, the step glaeda's job-started hook runs in, as a refusal at any length (hook_refused()). No workflow code runs in that step, and a failed hook skips every workflow step except the always() ones, so a failure there always belongs to the runner. A late "Set up job" failure still follows the 360 s rule. So does a hook that passed and was followed by a failing step.

Before: a hook refusal after more than 360 s ended red, with no re-run. After: once the run finishes, the rescue re-runs its failed jobs, the same as it already does for a quick refusal.

On the mini that refused this job, 9 of 26 waiter refusals in the last 24 h came after more than 360 s of waiting. All 9 were left red.

Tests

tests/test_ci_owned_pool_rescue.py (289 tests, all pass):

  • The real job's step shape at 770 s counts as refused, and assess() returns "refused". A failing always() step after it changes nothing. A passed hook followed by a failed step, a late "Set up job" failure, and a non-owned label are not refusals.
  • follow() re-runs the failed jobs once, through rerun-failed. With hook_refused patched out, which is the old behaviour, the same job gets no re-run.

Risk: low. The change only widens what counts as a refusal, and only for the runner's own hook step. Codex (gpt-6-astra, xhigh) reviewed it and returned SHIP with no findings.

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Fixes the owned-pool rescue leaving red jobs that glaeda's hook refused after a long capacity wait.

A failed Set up runner step, the step glaeda's job-started hook runs in, now counts as a refusal regardless of job length. Previously the rescue only recognized refusals within REFUSAL_SECONDS (360 s), so a hook refusal after a longer capacity wait was read as a code failure and never re-run. No workflow code runs in that step, so its failure always belongs to the runner.

Scope of the change

  • A late Set up job failure still counts only within REFUSAL_SECONDS.
  • A passed hook followed by a failed workflow step is still not a refusal.

Summary by CodeRabbit

  • Bug Fixes
    • CI jobs that fail while setting up a runner are now recognized as refused and can be retried, even after a long wait for capacity.
    • A later cleanup-step failure does not prevent a runner setup refusal from being recognized.
    • Failures in other setup or workflow steps, successful runner setup, and jobs on other runner providers are not classified as runner setup refusals. The existing handling of jobs with no completed steps is unchanged.