ci: owned-pool rescue never cancels a started nightly for a slow runner
What
The owned-pool rescue no longer cancels a started nightly (main nightly or nightly-next) because one runner is slow.
assess() counted only owned in_progress jobs as blockers (on_mini). So when build-nightly-app waited on the trusted pool, the rescue cancelled the whole run while a Blacksmith or Linux sibling was mid-job, such as the Ghostty CLI helper build. For a nightly (reserved=True), the rescue now never cancels the run while a sibling is still running:
- Queued job: every running job, on any pool, blocks the cancel. A stuck owned job waits until nothing else runs.
- Job held in setup: running siblings are not cancelled, not even as the watch ends.
- Refusal at the watch limit: if the nightly is still running other jobs, it is left alone with a warning instead of being cancelled.
- Right before any cancel in
rescue(), both the superseded-head cancel and the stuck-job cancel: the run's jobs are read again (started_nightly_work()). A nightly with any job running is not cancelled, because a sibling can start between the look and the cancel. The nightly-next tick re-runs a failed tip, and the next nightly builds main.
Nothing changes for other workflows.
Why
The rule is that a started nightly is never cancelled for a slow runner. The existing test Nightly.test_a_stuck_build_moves_to_blacksmith_keeping_what_passed encoded the bug: it cancelled a running Blacksmith helper.
Tests
tests/test_ci_owned_pool_rescue.py:
-
The old test is flipped so a stuck build never cancels a started sibling.
-
New tests: a stuck build moves once nothing else runs, a held build at the watch end, and a refusal at the watch end.
-
The refusal tests whose run finishes now let their helper finish too. Before, the helper never finished, and those tests only passed through the deadline cancel.
-
New tests for a Linux sibling that starts after the look (including a superseded run) and for a Linux sibling at the look.
The full file passes (257 tests).
Risk
Low. Only nightly runs change, and in the direction of cancelling less. A nightly whose owned job stays stuck while siblings run now waits instead of moving to Blacksmith mid-run.
Agent session: agent session
🤖 Generated with Claude Code
Summary by CodeRabbit
- Bug Fixes
- Nightly runs are no longer cancelled while jobs are in progress, including during runner setup or on other execution environments.
- If a refusal remains at the watch deadline while jobs are running, the nightly run is left untouched.
- Before cancelling a nightly run, the system checks for newly started jobs. If work is active, it resumes watching unless the deadline has passed. The check also applies when retrying a rate-limited cancellation, and cancellation stops if the run has finished or its attempt has changed.