ci(nightly-next): cancel a nightly or promotion run stuck on a runner label that stopped serving

Merged
TE
8d ago
+853-0

A nightly or promotion job can wait indefinitely after its runner label stops serving. Repository-variable changes do not move an already-created job. This makes the serialized nightly tick recover a run only when it has waited at least 20 minutes without a runner and an existing path will rebuild it.

The tick reads each run and all its jobs twice before canceling. Queue age starts no earlier than the latest executed job completion, because downstream records can predate their dependencies. The current nightly tip and its rebuild eligibility are revalidated immediately before each cancellation write. Running jobs, environment approvals, concurrency gates, jobs already assigned a runner, queued owned-pool jobs, unreadable API responses, and a changed run attempt keep the run intact. Only an aged queued job on non-owned labels identifies a stuck run; all other jobs must be completed or queued without a runner. Cancellation writes are sent once. A force-cancel requires another fresh stuck-state check on the same attempt. A newer commit alone never cancels a build.

Promotion recovery is limited to main and is disabled in train mode. The recovery step has a five-minute limit so later tick steps still run.

The tip's push run below the retry cap is rebuilt by the following cancelled-tip step; older stuck commits give way to the tip; a stuck promotion is requested again. A tip dispatch or a push at the retry cap is left queued with a named warning because this change cannot rebuild it.

This is the core recovery from #27095, refreshed onto main at 181a0680b20ea39bb930ba5295535ac214569519. No workflow permission or runner-trust policy changes are included. The 836 added lines keep one cancellation state machine and its behavioral tests together.

Observed / Not run

Observed on exact head f2dc2c0b61feb01291da72ac8da2659ff1572c20 (unchanged related-suite evidence retained from the prior head):

  • python3 tests/test_nightly_next_stuck_run.py: the regression-only commit 8c768aacec6 fails on premature downstream cancellation, changed-tip cancellation, and invalid rebuild paths; the later regression-only commit 43ceb593fdd fails three cases where an owned build is queued beside a downstream signer (discovery, before cancel, and before force-cancel); the final repair passes all 18 tests using a fake GitHub API and zero live cancellations.
  • test_nightly_next_cancelled_tip.py: 12 passed; test_nightly_next_rolling.py: 30 passed; test_nightly_promotion_admission.py: 88 cases passed; delivery concurrency passed.
  • test_gh_budget_wait.py: 18 passed; quota hold: 9 passed; runner label policy: 33 passed; off-Blacksmith tests: 7 passed, 1 skipped.
  • python3 scripts/verify-local.py: 16 static checks passed; Swift parsing skipped because no Swift files changed.
  • The landed unsigned-resume integration passes 21 tests; test_ci_owned_pool_sources.py passes with #27485.
  • Actionlint passed on the changed workflows and integrated feed watch; execution registry validation passed; git diff --check passed.

Not run: live cancellation, nightly dispatch, signing or publication. The cmux-next prepush-gate script does not exist on this main tree. Native compilation is unnecessary for these Python and workflow changes.

Changelog

none

Agent session: agent session