ci: preserve fresh rescue capacity behind failed recovery receipts
Summary
Ten unreadable pending cancellation receipts could consume every rescue slot on every minute pass, indefinitely starving a newer recoverable receipt and fresh stalled jobs. Recovery IDs now rotate, and old recovery attempts use at most half the per-pass attempt and settle budgets. Fresh stalled jobs retain the remaining capacity. Runs younger than the three-minute stall cutoff are also excluded from listing queries and the retained candidate cache, so they spend no job-read budget until they age into the window.
This fixes the P2 follow-up from the Astra review of #34820, building on fair discovery in #34807. Pending ownership, the durable cancellation journal and strict current-open-head checks are unchanged. The cursor is process-local and stops with the existing sweeper handover. The existing CI_HOSTED_LIGHT_RESCUE_DISABLED=1 switch disables the path.
Leo asked: "more things we can skip/do better" and "just build out everything, bro. We want everything. We want it all".
Performance
| Metric | Before | After | Delta | Harness or run |
|---|---|---|---|---|
| Fresh stalled job behind 10 permanently unreadable receipts | not served within 3 passes | rescued on pass 1 | starvation removed | RecoverCancelled.test_unreadable_owned_cancellations_do_not_starve_newer_recovery_or_fresh_work |
| Eleventh pending receipt behind the same 10 failures | not served within 3 passes | recovered on pass 3 | eventual service | same fixture |
| Premature job reads with 40 fresh runs and one stalled run | 39 | 0 | -39 reads/pass | Sweep.test_fresh_runs_spend_no_job_reads_and_become_eligible_after_aging |
| Total rescue-attempt budget per pass | 10 | 10 | 0 | focused suite |
Validation
The behavioral regression failed at f382d949b6d with neither eligible class served, then passed after the fix. Command: python3 -m unittest tests.test_ci_hosted_light_rescue.RecoverCancelled.test_unreadable_owned_cancellations_do_not_starve_newer_recovery_or_fresh_work.
The fresh-run regression failed in both listed and retained cases at c8c522667c3 with 39 premature reads, then passed at 12fec165848 with zero premature reads and successful rescue after aging. Command: python3 -m unittest tests.test_ci_hosted_light_rescue.Sweep.test_fresh_runs_spend_no_job_reads_and_become_eligible_after_aging.
Observed / Not run
Observed: the focused suite passes all 48 tests, including unreadable receipts, slow reads, cancellation ownership, current-head checks, scan fairness and job-read/mutation bounds. git diff --check passes. Fleet validation is running in run 41788953558 at source 12fec1658488d2a571c000178457b98256c1eb86; its terminal result will be posted before merge.
Not run: this fixture does not establish the production two-hour queue/quota acceptance window, which belongs to the coordinating lane.
Changelog
none
🤖 Generated with Codex
Agent session: agent session
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes hosted light rescue starvation where unreadable pending cancellation receipts could consume every rescue slot on each pass and block newer recoverable receipts and fresh stalled jobs. Recovery IDs now rotate across passes, old recovery attempts use at most half the per-pass attempt and settle budgets, and runs younger than the three-minute stall cutoff are excluded from listing queries and the retained candidate cache. Fixes #34931 and #34990.
Summary by CodeRabbit
- Bug Fixes
- Stalled runs are now considered for recovery only after they meet the required age, preventing newer runs from being processed too early.
- Recovery continues to service other eligible runs when an earlier recovery receipt cannot be read.
- Slow recovery attempts no longer prevent fresh stalled runs from being handled within the same pass.