base red: release-notary RunsOnFallbackTests
Summary
This fixes a base red, priority zero. nightly-guards / workflow-guard-tests / release-notary fails on every feat-cmux-next push and PR (for example #28849, #28916, #29024, #28958). The job has three separate causes. Each one hides behind the one before it in the same step, so this PR fixes all three:
RunsOnFallbackTests, two tests. The first red sha is a7b6a61175c ("legacy sign and deltas run on owned Macs only"); its parent is green. That commit endedsign-nightly-identity-legacy,verify-nightly-identity-legacyandgenerate-nightly-deltasonvars.CI_SIDE_LANE_RUNNER. That is a reserved lane: #28249 forbids a chain that ends on a reserved variable, because an unset or offline lane leaves one dead queue. The commit also removed the legacy sign's picked Blacksmith label.- Workflow fix: each of the three chains now ends
|| vars.CI_NIGHTLY_NEXT_BUILD_RUNNER || vars.CI_SIDE_LANE_RUNNER || 'glaeda-std-xcode-26.6'.glaeda-std-xcode-26.6is the owned std pool (42 slots invars.CI_OWNED_POOL_SLOTS), on the same minis as the side lane. The jobs stay on owned Macs only and use no Blacksmith label (Lawrence, 2026-10-30). The chains do not end on a reserved variable. Today both variables are set, so placement does not change. The literal is used only when an operator clears both variables. - Test: the test keeps the Blacksmith re-run alternate rule for
build-sign-notarize-nightly. The legacy sign and its verify no longer read a Blacksmith pick, so a new test applies the owned form of the same rule to them: an owned pick holds only on the attempt that picked it, so a re-run of failed jobs never queues on that Mac. I did not add an exemption. The exact-string pins from a7b6a61175c (test_nightly_sign_runner.py,test_ci_nightly_off_blacksmith.py,test_nightly_universal_build.sh) now pin the new ending.
- Workflow fix: each of the three chains now ends
test_ci_warm_sources. It runs in the same step after the runs-on test, so it was hidden. 13 jobs drifted from.github/warm-sources.json: stable-next.yml (#27898), chromium staleness, acpmux tests, the test-compile plan, the cmux-tui cargo caches, and the Blacksmith twin that a7b6a61175c removed. The four cmux-tui lines are the same as in #28818, so the two PRs merge clean.SigningHooksInStepWithGlaedaTests. This cause is outside the repo. glaeda 1cd373c3 (teamleaderleo/glaeda#1768, 06:22Z) released runner hook49b5d9f5. I checked that sha256 ofscripts/glaeda-cmux-runner-hookon glaeda main is49b5d9f59f58…, and that glaeda-disk pins it as its newest hook. The hook diff after 485d1dcc lengthensSIGNING_DRAIN_WAIT_Sto 3600 s and admits a handed job beside a fleet waiter. The exclusive claim and the leftover-process checks do not change. The line is the same as the one the sync PR #29029 adds.
PR #28967 tries to fix cause 1 with a test exemption. That makes the test weaker, and it does not fix causes 2 and 3 (its own release-notary check is red on the warm-sources test).
Risk: high. This PR changes the runs-on of a signing job (sign-nightly-identity-legacy) and the list of trusted signing hooks. It does not change triggers or permissions.
Testing
Focused command: python3 tests/test_ci_runs_on_fallback.py
- On the base (7ad4616221f; first red a7b6a61175c, green at a7b6a61175c^):
FAILED (failures=2). - On the head:
Ran 20 tests ... OK.
python3 tests/test_ci_warm_sources.py: 13 errors on the base, OK on the head. python3 tests/test_ci_signing_owned_mac.py: fails on the base with "glaeda's newest hook 49b5d9f5 is not in glaeda-signing-hooks.txt", OK on the head.
Negative checks on the head: without the attempt guard on the owned pick, the new test fails. Without the pool literal, test_no_runs_on_chain_ends_on_the_reserved_side_label fails.
Observed / Not run
- Ran on a Linux VM: every
release-notarystep of ci-guards.yml, at the head after a rebase onto 7bda8ef95ec. All pass, except two steps that fail only on this VM: app bundle license compliance (noghostty-nextsubmodule here) and the arm64-only track (nolipoon Linux). Both pass in CI on the base (run 41164501878). - Ran:
python3 scripts/cmux-next/prepush-gate.py, which reported no new violations. - Not fixed here:
tests/test_runner_routing_table.py(groupci, not release-notary) is red on the base too. Its causes are stable-next.yml jobs that are not inrunners.jsonandgenerate-nightly-deltas, which reads the side lane with no placement output. This PR does not make it worse. - Not run: a real nightly run on the changed chains. Both variables are set, so the new literal is not used today.
Changelog
none
Proof
CI only: no UI change.
![View with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png)
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes the release-notary CI job, which is red on every feat-cmux-next push, by addressing three separate causes.
- The legacy sign, its verify, and nightly deltas now end their
runs-onchain on an owned std pool literal (glaeda-std-xcode-26.6) instead of a reserved side lane variable; placement is unchanged while both lane variables are set. An owned pick now holds only on the attempt that picked it, so re-runs of failed jobs never queue on a stalled Mac. - Updates
warm-sources.jsonfor 13 jobs that drifted from it, including the removal of the Blacksmith twin froma7b6a61175c. - Pins glaeda runner hook
49b5d9f5as trusted; the hook lengthens the signing drain wait without changing claim or leftover-process checks.
Risk: high — this changes the runs-on of a signing job and the trusted signing hooks list, but not triggers or permissions.