ci: resume hosted-light cancellations after rerun errors
A hosted-light rescue that cancels a stalled run now retains its ownership receipt when the rerun request fails. It retries transient failures after checking the live attempt and current open PR; an accepted rerun with a lost response is detected before another POST. Pending receipts survive later minute passes and may be restored from complete finished sweeper logs. Unmarked cancellations and incomplete history never authorize recovery.
Fixes #34810. This builds on merged #34571, adds no second mutation owner, and uses the workflow token. The existing CI_HOSTED_LIGHT_RESCUE_DISABLED=1 switch stops the worker; reverting this patch restores the previous behavior.
Leo asked: “how’s CI? we still have bigger wins we can get? what’s the state of it all? more things to skip/do better”.
Performance
| Metric | Before | After | Delta | Harness |
|---|---|---|---|---|
| Successful recovery after one transient rerun 502 | 0/1 | 1/1 | +1 recovered run | python3 tests/test_ci_hosted_light_rescue.py RecoverCancelled.test_a_transient_rerun_error_recovers_without_another_cancel, fleet run 41784181549 |
The retry loop permits at most three attempts within the existing shared 90-second pass budget; handover stops the worker. History restoration reuses the bounded journal reader and fails closed when capped. The narrow race with an external manual rerun between the last live read and POST remains possible. Persistent pending failures consuming every slot are tracked in #34931.
Observed / Not run
The focused command failed at regression commit ba7dc190eb83761b19a33ff96cbfaa65ff926121 and passed with the fix. Prior fleet run 41774627366 tested source 8c452b127b28a92624a40661023bbbf4ef022171; the hosted-light suite (41 tests), owned-pool suite (468), hosted-runner guard and workflow-wiring check passed. The final integrated head is 0ea8e7c4ecd1ccc384e1b35d077e24a7f66905c7; fleet run 41784181549 passed all 555 focused tests. The successful run verified snapshot f6733685a68bc683fb1d379cab26ff7e261ec558 is its sole child with the identical source tree 7a1fa9b07d6b214958a4c7650213907a85291f64. git diff --check passes.
Independent Astra review: SHIP at 0ea8e7c4ecd1ccc384e1b35d077e24a7f66905c7, with follow-up #34931. No production cancellation was induced, and this fixture does not establish the two-hour queue acceptance window. No app build is needed.
Changelog
Fixed: resume owned hosted-light cancellations after transient rerun errors.
🤖 Generated with Codex
Agent session: agent session