base red: release-notary RunsOnFallbackTests

LA
7d ago
+53-20

Summary

This fixes a base red, priority zero. nightly-guards / workflow-guard-tests / release-notary fails on every feat-cmux-next push and PR (for example #28849, #28916, #29024, #28958). The job has three separate causes. Each one hides behind the one before it in the same step, so this PR fixes all three:

  1. RunsOnFallbackTests, two tests. The first red sha is a7b6a61175c ("legacy sign and deltas run on owned Macs only"); its parent is green. That commit ended sign-nightly-identity-legacy, verify-nightly-identity-legacy and generate-nightly-deltas on vars.CI_SIDE_LANE_RUNNER. That is a reserved lane: #28249 forbids a chain that ends on a reserved variable, because an unset or offline lane leaves one dead queue. The commit also removed the legacy sign's picked Blacksmith label.
    • Workflow fix: each of the three chains now ends || vars.CI_NIGHTLY_NEXT_BUILD_RUNNER || vars.CI_SIDE_LANE_RUNNER || 'glaeda-std-xcode-26.6'. glaeda-std-xcode-26.6 is the owned std pool (42 slots in vars.CI_OWNED_POOL_SLOTS), on the same minis as the side lane. The jobs stay on owned Macs only and use no Blacksmith label (Lawrence, 2026-10-30). The chains do not end on a reserved variable. Today both variables are set, so placement does not change. The literal is used only when an operator clears both variables.
    • Test: the test keeps the Blacksmith re-run alternate rule for build-sign-notarize-nightly. The legacy sign and its verify no longer read a Blacksmith pick, so a new test applies the owned form of the same rule to them: an owned pick holds only on the attempt that picked it, so a re-run of failed jobs never queues on that Mac. I did not add an exemption. The exact-string pins from a7b6a61175c (test_nightly_sign_runner.py, test_ci_nightly_off_blacksmith.py, test_nightly_universal_build.sh) now pin the new ending.
  2. test_ci_warm_sources. It runs in the same step after the runs-on test, so it was hidden. 13 jobs drifted from .github/warm-sources.json: stable-next.yml (#27898), chromium staleness, acpmux tests, the test-compile plan, the cmux-tui cargo caches, and the Blacksmith twin that a7b6a61175c removed. The four cmux-tui lines are the same as in #28818, so the two PRs merge clean.
  3. SigningHooksInStepWithGlaedaTests. This cause is outside the repo. glaeda 1cd373c3 (teamleaderleo/glaeda#1768, 06:22Z) released runner hook 49b5d9f5. I checked that sha256 of scripts/glaeda-cmux-runner-hook on glaeda main is 49b5d9f59f58…, and that glaeda-disk pins it as its newest hook. The hook diff after 485d1dcc lengthens SIGNING_DRAIN_WAIT_S to 3600 s and admits a handed job beside a fleet waiter. The exclusive claim and the leftover-process checks do not change. The line is the same as the one the sync PR #29029 adds.

PR #28967 tries to fix cause 1 with a test exemption. That makes the test weaker, and it does not fix causes 2 and 3 (its own release-notary check is red on the warm-sources test).

Risk: high. This PR changes the runs-on of a signing job (sign-nightly-identity-legacy) and the list of trusted signing hooks. It does not change triggers or permissions.

Testing

Focused command: python3 tests/test_ci_runs_on_fallback.py

python3 tests/test_ci_warm_sources.py: 13 errors on the base, OK on the head. python3 tests/test_ci_signing_owned_mac.py: fails on the base with "glaeda's newest hook 49b5d9f5 is not in glaeda-signing-hooks.txt", OK on the head.

Negative checks on the head: without the attempt guard on the owned pick, the new test fails. Without the pool literal, test_no_runs_on_chain_ends_on_the_reserved_side_label fails.

Observed / Not run

  • Ran on a Linux VM: every release-notary step of ci-guards.yml, at the head after a rebase onto 7bda8ef95ec. All pass, except two steps that fail only on this VM: app bundle license compliance (no ghostty-next submodule here) and the arm64-only track (no lipo on Linux). Both pass in CI on the base (run 41164501878).
  • Ran: python3 scripts/cmux-next/prepush-gate.py, which reported no new violations.
  • Not fixed here: tests/test_runner_routing_table.py (group ci, not release-notary) is red on the base too. Its causes are stable-next.yml jobs that are not in runners.json and generate-nightly-deltas, which reads the side lane with no placement output. This PR does not make it worse.
  • Not run: a real nightly run on the changed chains. Both variables are set, so the new literal is not used today.

Changelog

none

Proof

CI only: no UI change.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the release-notary CI job, which is red on every feat-cmux-next push, by addressing three separate causes.

  • The legacy sign, its verify, and nightly deltas now end their runs-on chain on an owned std pool literal (glaeda-std-xcode-26.6) instead of a reserved side lane variable; placement is unchanged while both lane variables are set. An owned pick now holds only on the attempt that picked it, so re-runs of failed jobs never queue on a stalled Mac.
  • Updates warm-sources.json for 13 jobs that drifted from it, including the removal of the Blacksmith twin from a7b6a61175c.
  • Pins glaeda runner hook 49b5d9f5 as trusted; the hook lengthens the signing drain wait without changing claim or leftover-process checks.

Risk: high — this changes the runs-on of a signing job and the trusted signing hooks list, but not triggers or permissions.