ci(cmux-next): DEV signer on Blacksmith 6 vCPU, no evicting sign group

Merged
TE
11d ago
+24-7

The DEV feed (cmux NEXT DEV, cmux-next-dev-daily.yml) has published nothing since 00:13Z on 2026-10-26, more than six hours against the 30-minute freshness rule. The DEV compiles that feed it keep succeeding; the publisher's sign job is what stalls:

  • Runner. Its only label is blacksmith-12vcpu-macos-26. Run 40258137182 queued at 02:14Z, got a runner at 04:50Z, and then failed on the installation rate limit. Run 40268668036 has been queued since 04:21Z. Nightly's own build-nightly-app jobs were queued on the same label at the same time.
  • Eviction. The sign job's cmux-next-dev-daily-sign group keeps one pending run and replaces it with each newer handoff. So while one signer waited for a runner, every newer source run was cancelled before it started (40263045581, 40268026359, 40268457326, 40278704661).

Change

  • sign moves from blacksmith-12vcpu-macos-26 to blacksmith-6vcpu-macos-26, the label verify and tip already use. The 12 vCPU label runs about five jobs at once for the whole organization, and it held this signer for hours while 6 vCPU (about ten slots) started the tip job within minutes. Signing needs no large runner. Hosted macos-26 was the first try, but the runner guards refuse it outside the fork branch: a billing block must not stop CI, and macos-26 is also a fleet label (tests/test_ci_self_hosted_guard.sh).
  • sign has no concurrency group. Signing shares no state between source runs: the build number is the source run id, and the artifacts belong to each run. publish refuses to roll the prerelease back (not_newer), and tip refuses to roll the feed back (tip_feed.py previous). Both keep their single-writer groups. A newer handoff now signs alongside an older one instead of being evicted by it.
  • scripts/ci/check_workflow_concurrency.py allowlists this publisher and says why. The dispatcher sends at most one run per finished source run, at most one every five minutes.

Not changed: the installation's REST quota, which this run also hit, is a separate drain; fanout-infra owns it. Owned Macs are also allowed for DEV signing under Leo's standing rule. This PR stays on the Blacksmith label the guards accept.

Tests: tests/test_cmux_next_dev_tip_feed.py now checks that sign and tip never queue on the 12 vCPU label, that there's no sign group, and single writers for publish and tip. These pass locally: test_ci_self_hosted_guard.sh, test_ci_workflow_concurrency.py, test_ci_fork_runner_routing.py, the three signing-secret and keychain policy tests, and the dispatcher test.

Risk: medium. This is the DEV publisher's runner. Checking: the next dispatched publisher run's sign job starts within minutes on 6 vCPU, and cmux-next-dev shows a newer commit.

Agent session: cosmic-gray

🤖 Generated with Claude Code

Summary by CodeRabbit

  • CI Improvements
    • Daily development handoffs are no longer serialized or superseded by a shared concurrency group.
    • Signing jobs use updated macOS runners, while tip and publish jobs retain their concurrency controls.