ci: preserve started nightly work during runner stalls

Merged
TE
6d ago
+99-24

Summary

The nightly-next stall rescue cancelled run 41269080663 after decide, signing and failure-report work had already started, because later jobs waited on an unavailable runner. This PR keeps any run with recorded started work intact, while still cancelling never-started stalls and jobs that timed out, and makes the tick's cancellation step unbuffered with enough timeout for its bounded settle sequence.

Leo: "IF YOU DONT MERGE STUFF LIKE THAT WITHIN THE DAY THERE'D BETTER BE AN EXTREMELY GOOD REASON" and "WITHIN 3 HOURS IF NEEDS CHANGES / WITHIN THE HOUR IF STRAIGHTFORWARD".

Changes

  • Parse each job's started_at and protect a run when a non-skipped job started or is in progress.
  • Report protected stalls as started, waiting on a runner so capacity problems remain visible.
  • Keep the certain-to-fail timeout exception: a started job with timed_out still permits cancellation.
  • Run the rescue with python3 -u and raise its step timeout from 5 to 15 minutes, covering the settle and force-cancel reads.

Tests

tests/test_nightly_next_stuck_run.py covers a started run with a queued later job, a never-started stuck run, and a timed-out started job. The workflow assertion covers the unbuffered command and extended step timeout.

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Stops the nightly-next stall rescue from cancelling runs that already started work, avoiding the loss of progress like the decide, signing, and failure-report stages of run 41269080663 when later jobs waited on an unavailable runner.

  • Parses each job's started_at and keeps a run when a non-skipped job started or is in progress; a timed-out started job still permits cancellation since it is certain to fail.
  • Reports protected stalls as "started, waiting on a runner" so capacity problems remain visible.
  • Runs the rescue unbuffered (python3 -u) and extends its step timeout from 5 to 15 minutes.

Summary by CodeRabbit

  • Bug Fixes
    • Nightly runs with work already underway are less likely to be canceled while waiting on queued jobs. Runs may still be canceled when started work times out.
    • Queued jobs without an available runner are distinguished in stuck-run status updates, clarifying why a run is waiting.
  • Chores
    • The stuck-run cancellation step now has a 15-minute timeout, and its output appears sooner in workflow logs.