ci(nightly): the legacy-identity feed compare takes the signer route, not a 3-minute fleet wait

Merged
TE
5d ago
+33-112

The cmux NIGHTLY-identity copy (the old feed, files-next.cmux.com/nightly-next/appcast-arm64.xml, which existing com.cmuxterm.app.nightly installs read) starts with legacy-identity-target, a 14-20 s feed compare that picks the build the signer copies. Its runner came from pick-legacy-identity-runner, which ran portable_job_placement.py with NIGHTLY_WAIT_FOR_FLEET=1 and LINUX_ONLY on cmux-fleet-linux-x64. In every run on 2026-11-02 that placement waited the full three minutes, reading the runner list every 30 s (about 28 API calls), found no idle fleet Linux runner, and sent the job to blacksmith-4vcpu-ubuntu-2404 anyway.

What changed:

  • legacy-identity-target uses the signer route that signer-app, signer-app-notary and signer-dmg already use: GitHub-hosted Ubuntu first, Blacksmith Linux 4 vCPU on a re-run. The runs-on line is identical, so it falls under the hosted guard's existing exact-line exception (tests/test_ci_self_hosted_guard.sh). The job picks the signing job's build, holds no Apple secret, and keeps release-next only for the release App token. Hosted and Blacksmith runners already run that environment's signing jobs, so no trust, secret, environment or permission changes.
  • pick-legacy-identity-runner is removed, along with its hosted_light_jobs entry and step coverage. Its if: moves onto the target unchanged, and the target still runs after publish-nightly. The order from #25957 is kept.
  • Tests: test_ci_nightly_off_blacksmith now checks the signer route instead of the picker, and the job joins the signer jobs in the documented Blacksmith-only debt list (the rulebook's measured hosted exception). test_nightly_trusted_routing treats it as a signer-route job. Picker references are removed from test_nightly_cmux_tui_resolve and test_nightly_sign_runner. The linux_sizes entry drops its 8 vCPU failover.

Performance

Legacy-feed lag behind cmux NEXT, measured from the end of publish-nightly to the start of legacy-identity-target:

metricbeforeafterdeltaharness or run link
publish-nightly end → legacy-identity-target start3.3-4.2 min (8 runs)~0.2 min expected (hosted start; signer-build started 12 s after its picker in run 41416268105)about -3.2 min per runruns 41412703950, 41413478144, 41414248977, 41415054395, 41416268105, 41417028889, 41417840707, 41418619409
pick-legacy-identity-runner wall time3.2-3.4 minjob removed-3.3 minsame runs
runner-list reads per run for the target~280-28same runs
Old-feed lag behind the NEXT feed, end to end (run 41417028889, no runner wait)21.5 min (publish-nightly 09:42:17 → old feed 10:03:48)~18.3 min expected-3.2 min41417028889
Blacksmith Linux minutes per run for the target~0.3 min (4 vCPU)0 on attempt 1-0.3 minsame runs

Most of the remaining lag is in sign-nightly-identity-legacy, which needs an exclusive owned Mac. In these runs, its "Set up runner" step (waiting for glaeda's exclusive admission) took 12-33 min, or was still waiting. A follow-up PR addresses that.

Validation, run locally:

  • tests/test_ci_nightly_off_blacksmith.py, test_nightly_trusted_routing.py, test_nightly_cmux_tui_resolve.py, test_nightly_sign_runner.py, test_ci_portable_job_placement.py, test_ci_runs_on_fallback.py, test_nightly_next_post_publish.py, test_runner_routing_table.py, test_nightly_next_feed_watch.py, tests/test_ci_self_hosted_guard.sh: pass.
  • All 131 tests that read nightly.yml, runners.json or the step coverage were run. Six fail, and each fails the same way on base: test_check_ghostty_zig_workflows, test_ci_blacksmith_linux_failover, test_ci_blacksmith_linux_sizes (cloud-newlink-e2e.yml#create-and-link), test_ci_cloud_probe_rescue, test_ci_linux_single_size, test_nightly_next_arm64_only.

Prior art: #30471 routed this job through fleet Linux, and in practice that sent it to Blacksmith after three minutes every time. #31831 put the signer jobs on hosted Ubuntu first. #25957 (post-publication order) is unchanged.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Routes the nightly legacy-identity feed compare (legacy-identity-target) onto the signer route, cutting about 3 minutes of dead wait per run.

The legacy-identity-target job (a 14–20 s feed compare that picks the build the signer copies) previously waited on pick-legacy-identity-runner, whose fleet-Linux placement polled for an idle runner for the full three-minute budget and then sent the job to Blacksmith anyway. It now runs on GitHub-hosted Ubuntu first, falling back to Blacksmith Linux 4 vCPU on a re-run, same as signer-app and signer-dmg. The picker job is removed entirely.

Key changes

  • legacy-identity-target keeps publish-nightly in its needs and runs after it, preserving the post-publication order.
  • The job holds only the release App credentials already used by the signer route; no new secrets, environments, or permissions.
  • Tests updated to assert the signer route, move legacy-identity-target into the documented Blacksmith-only debt list, and drop picker references.