ci(nightly-next): an older build hands its sign slot to a newer build whose sign job is queued

Merged
TE
2d ago
+323-1

An older nightly-next build hands its sign slot to a newer build whose sign job is already queued, running or signed. Nothing is cancelled, and the newest finished build always signs.

What was wrong

GitHub gives a free runner the oldest queued job on its label, and a runner can't choose another. Publication is latest-wins. So on 2026-11-04 around 17:25Z, the three sign slots were held by the oldest nightly-next builds (runs 41866781601, 41867447845, 41868557041), and the newer builds' build-sign-notarize-nightly jobs waited behind them: 15, 19 and 24 minutes for the three that had started, with three more still queued (41869470224, 41870869803, 41873130397). Each older build then published a nightly that the next one replaced within minutes, and the newest tip reached the feed about an hour after its landing.

The existing "Leave notarization to a newer published build" check (sign-pick-overtaken.sh, #33109) runs in pick-nightly-sign-runner, before the sign job joins the queue. With oldest-first order it almost never fires.

What this changes

  • One step, Yield the sign slot to a newer build, runs in build-sign-notarize-nightly right after the claim, before any secret or signing work. It is scripts/ci/nightly_sign_yield.py, run from the trusted signing controls.
    • It reads the in-progress nightly-next push runs newer than this one.
    • If one has its own sign job queued, running or signed, this job exits 1 with yielded=true, and the runner takes the next queued job (about a minute instead of a full sign and notarize).
    • Otherwise, and on any doubt (an unreadable answer, a malformed field, a listing that doesn't add up), it signs as before.
    • Scope: attempt 1 of a nightly-next push only. Re-runs, dispatches and main's nightly never yield. Setting NIGHTLY_NEXT_SIGN_YIELD=0 turns it off.
  • report-nightly-failure does not report a yielded sign whose run had nothing else fail. This is the same pattern as its existing "replaced" build.
  • Nothing is cancelled, and no concurrency group changes. The rulebook's "never cancel a started NEXT nightly" holds. Verify, publish and deltas are skipped for the yielded run because the sign job did not succeed, as for any failed sign.

The rule this narrows (#33109)

sign-pick-overtaken.sh says: "Only a published newer build counts, never a newer push alone: a newer run that then fails must not leave the feed with nothing (#28961)."

  • What still holds: a newer push alone still never counts. A newer run whose build is still running, or whose sign job failed, was skipped or was cancelled, is no reason to yield.
  • What now counts: a newer run whose sign job exists, which GitHub creates only once that run's build jobs pass.
  • Why it can't starve the feed: the newest such sign job never yields (no newer run has one), so the newest finished build always signs. #28961's starvation came from a run skipping its build as soon as nightly-next moved, before any newer build existed.
  • What is given up: if that newest sign then fails, the older builds that yielded to it go unsigned, and the next tip builds as usual.

Decided on cmuxterm-hq#7079. Ordering the slots newest-first was tried first. teamleaderleo/glaeda#1815 found that a runner can't choose which queued job it takes; glaeda can only refuse a job, which fails it.

Tests

tests/test_nightly_sign_yield.py (new, linux-guard):

  • the 2026-11-04 shape: the newest queued build never yields, nor does a newer run whose sign job doesn't exist yet;
  • an older build yields to a newer sign job that is queued, running or signed;
  • a newer push alone never counts (no sign job, a failed, skipped or cancelled one, or the legacy-identity signer);
  • an older run is never a reason;
  • out of scope (attempt 2, dispatch, main, another track, NIGHTLY_NEXT_SIGN_YIELD=0, a bad id): it signs without a read;
  • any doubt signs;
  • the CLI writes yielded=false and exits 0 when the API is unreachable;
  • wiring: the step sits right after the claim, with no secret before it or in it, and report-nightly-failure skips only a yielded sign.

Red on base: the wiring tests fail on 4d20a29a816 (no such step or output); the decision tests have no script to load there. Green on this head.

python3 -m unittest tests/test_nightly_sign_yield.py tests/test_nightly_step_coverage.py: 14 OK on this head. Every other test file that reads nightly.yml (106) was run on the head and on 4d20a29a816: 69 pass on both, and the remaining 37 fail exactly the same tests on both (they need tools this checkout lacks, such as actionlint).

Observed / Not run

  • Observed: the run and job timings above, read from the Actions API at about 17:25Z.
  • Not run: a live nightly-next run with the step. The first nightly-next runs after this lands show it in their sign job's log (signing: ... or the Sign yield notice).

Follow-up, not here: a superseded run's sign-nightly-identity-legacy also takes the exclusive class on its mini (35 min on run 41799796683); the same yield could apply to it.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

An older nightly-next build now hands its sign slot to a newer build whose sign job is queued, running, or signed, cutting the newest tip's feed delay from roughly 20 minutes per slot to about a minute.

GitHub gives a free runner the oldest queued job on its label, so the newest build's sign job waited behind older builds whose nightlies were replaced within minutes. A new step right after the signing claim, before any secret, checks the in-progress nightly-next runs; if a newer run's sign job already exists (GitHub creates it only once that run's build passes), this job exits 1 and the runner picks the next queued job. Nothing is cancelled, and report-nightly-failure skips a yielded sign whose run had nothing else fail.

  • Narrowing #33109's rule: a newer push alone still never counts, but a newer run whose sign job exists now does. The newest sign job never yields, so the newest finished build always signs; the trade-off is that if that sign then fails, the older builds that yielded to it go unsigned.
  • The yield script pages the run and job listings until an empty page, since a status-filtered list's total_count can run ahead; more than MAX_PAGES pages signs as before. The test is registered as manual, so the registry validator passes.