ci: schedule hosted light rescue independently
Hosted light-job recovery now runs every minute inside the existing owned-pool sweeper with its own workflow token. A separate five-minute schedule wakes that single mutation owner, so a queued overflow probe or exhausted routing-App read budget cannot suppress light-job recovery.
Leo asked: “how’s CI? we still have bigger wins we can get? what’s the state of it all? more things to skip/do better”.
The optional route-policy checkout is limited to two minutes and may fail without stopping the established sweep. Recovery keeps the current-head, first-attempt, runner-unassigned and unfinished-sibling safeguards. The worker stops with sweeper handover and never overlaps its own passes. Transient cancel/rerun recovery is tracked in #34810.
Performance
| Metric | Before | After | Delta | Harness |
|---|---|---|---|---|
| Independent recovery wake opportunities/hour | 6, behind the ten-minute overflow probe | 12, on its own five-minute schedule | +100% | Parsed schedules in ci-cloud-overflow-probe.yml and ci-hosted-light-rescue.yml; tests/test_ci_runner_pools.py Declared |
While the sweeper runs, its hosted-light worker has a one-minute cadence independent of primary App pacing. This is a scheduling measurement; the production queue-duration acceptance window is not established by the fixture.
The feature is on by default. Set CI_HOSTED_LIGHT_RESCUE_DISABLED=1 to stop it; removing the new workflow and worker integration is the revert path.
Observed / Not run
Fleet run 41774475640 passed at source 08e8dc1fdaf03687cb04c15bbda36119386982e0: hosted-light (33 tests), owned-pool (468), runner-pools (45), runner-policy shell guard, workflow wiring and execution registry. The same run restores the old wiring, observes both named failures, then restores the final files and passes both checks. The pre-existing unregistered signer-parity test is a warning.
The former final-head CI failure was this PR's missing test wiring; this head fixes it. No app build is needed for these portable CI scripts. The new production schedule has not executed from this branch.
Changelog
Changed: recover hosted light-job queues independently from overflow probes and routing-App pacing.
🤖 Generated with Codex
Agent session: agent session
Summary by CodeRabbit
- New Features
- Added automatic recovery for eligible stalled hosted-light CI jobs, with scheduled and manual runs.
- Integrated hosted-light recovery with the owned-pool sweeper, with safeguards to avoid retrying jobs when a pull request or run is no longer eligible.
- Added a manual dry-run option to preview recovery actions without making changes.
- Bug Fixes
- Removed hosted-light recovery from the overflow-probe workflow; runner-pool switching and cancellation behavior remain.