ci(cmux-tui): the cmux_next_ daemon tests run on Blacksmith Linux 32 vCPU

Merged
TE
10d ago
+284-414

Leo, 2026-10-26: "but we have the fleet we have linux blacksmith... why are we STILL TALKING ABOUT BLACKSMITH MACOS"

Leo, 2026-10-26: "I THOUGHT THESE COULD JUST GO ON LINUX"

What changes

The nightly-next promotion gate, cmux-tui-artifacts.yml job cmux-next-daemon-tests (cargo test <members> --locked cmux_next_), now runs on Blacksmith Linux on every route. Before, it took the promote pool or CI_NEXT_DAEMON_POOL on attempt 1 of a feat-cmux-next push, CI_PIN_DAEMON_POOL or 16 vCPU Linux on pin pushes, and the owned side lane (CI_SIDE_LANE_RUNNER) or Blacksmith macOS everywhere else.

  • The gate: runs-on is blacksmith-32vcpu-ubuntu-2404, or ubuntu-24.04 in forks. Every event and every attempt. The macos_runner dispatch input and the pool variables no longer move it. It is renamed cmux_next_ daemon tests (Linux).

  • The marker: publish-tree-tested writes tree/<key>/daemon-tests.json after a pass on any OS. The old runner_os == 'macOS' condition is gone. cmux-tui-mark-daemon-tested.sh gets the job's runner_os as TESTED_OS and records it as "os". request-nightly-next.sh only checks that the marker exists, so no reader changes.

  • The publisher deferral: DAEMON_POOL in tree-preflight and tree-owner-wait is the gate's own label now, not vars.CI_NEXT_DAEMON_POOL || vars.CI_NEXT_PROMOTE_POOL. A publisher run whose gate is still queued on one of the retired Mac pools is not waited for. cmux_tui_publisher_testing.py finds the gate by its cmux_next_ daemon tests name prefix, so runs from before this change, whose job is named (macOS), still count.

  • New macOS shadow workflow: .github/workflows/cmux-next-daemon-macos-shadow.yml (cmux_next_ macOS-only daemon tests (shadow)) runs the tests that need macOS itself:

    • acpmux's Keychain and Seatbelt tests: no_keychain, router_no_keychain, remote_sandbox and remote_sandbox_canary;
    • cmux-tui's idle_wakeups, whose wakeup counter reads proc_pid_rusage only on macOS.

    It is a separate workflow, triggered by feat-cmux-next pushes that touch cmux-tui/**, ghostty-next or its own files, and by workflow_dispatch. No workflow_run. Nothing waits for it, so a queued or red shadow never holds the artifacts run open. Inside that run, it would stall the tree owner's failed-publication recovery. It has permissions: {} with contents: read on the job. One concurrency group per ref with cancel-in-progress: false allows one running and one waiting.

    It runs on blacksmith-6vcpu-macos-15 (macos-26 in forks). Its runs-on reads no variables or inputs, so it never takes an owned pool, which owned_pool_rescue doesn't watch for this workflow. CI_NEXT_DAEMON_POOL and CI_NEXT_PROMOTE_POOL no longer route any daemon-test job.

  • #25784's Mac picker is removed: this PR merges the base and deletes cmux-next-daemon-tests-runner, the job #25784 added to pick a free owned Mac or Blacksmith macOS for the gate. The gate's needs goes back to [tree-preflight, tree-owner-wait], and the picker's test assertions go too. The Linux route is the only one.

  • runners.json and the Xcode guard: the cmux-next-daemon-tests Mac class entry is replaced by the shadow workflow's job (macos-runtime-tests). test_ci_macos_xcode_selection.py exempts the Rust-only shadow and drops the gate's entry.

  • Tests:

    • test_cmux_tui_daemon_tests_linux.py and test_ci_tui_routes_avoid_nightly_label.py now check the Linux route under every pool variable, the marker's any-OS condition and os field, and that the shadow gates nothing and stays off the promote pool.
    • test_cmux_tui_publisher_testing.py checks the name-prefix match and that the shadow's name never matches the gate.
    • test_tui_publish_workflow_security.py pins the gate's Linux route and checks the shadow never takes the nightly label.

Takes back #25784: its Mac picker for the nightly-next daemon-test gate is replaced by the Linux-only gate (sr2's order: only the Linux gate stays). Run 40424832811 passed 284 tests in about 6 min on Blacksmith Linux.

Why

The tests the gate runs are Rust with no Xcode, AppKit, codesign or simulator. cmux-tui.yml's Linux job already runs the whole workspace. On the owned Macs the gate queued 20+ minutes on side-std (2026-10-26), and on the promote pool it queued behind the Release compile. Nightly-next waited either way. #23837 measured the same tests on blacksmith-32vcpu-ubuntu-2404 in run 39692110137: 280 passed, 0 failed, 8 min 20 s including a cold build.

Before: the promotion gate waited for an owned Mac (promote pool or side lane). After: it takes Blacksmith Linux 32 vCPU at once, and only a non-gating shadow uses a Mac.

Risk

Medium. This changes what gates nightly-next.

  • macOS-only coverage leaves the gate: the Keychain, Seatbelt and wakeup-counter tests no longer block promotion. They run in the shadow, and a red there is visible but blocks nothing. A macOS-only regression in those paths could reach a nightly.
  • Pin pushes now write the marker: a pin's Linux pass marks the tree, so the later feat-cmux-next push of the same tree key skips its tests. The tree key covers every input of the tests.
  • Retired pool variables: setting CI_NEXT_DAEMON_POOL or CI_NEXT_PROMOTE_POOL no longer moves the gate back to a Mac. To roll back, revert this PR.
  • Blacksmith Linux capacity: the gate now depends on Blacksmith Linux 32 vCPU capacity.

Follow-ups

  • docs/ci-runners.md:891 still describes the old Mac route and says Linux never writes the marker. That file is #24941's, so its owner updates it.

Signals

  • Success:
    • The next feat-cmux-next push's cmux_next_ daemon tests (Linux) job runs on blacksmith-32vcpu-ubuntu-2404 and goes green.
    • mark the published tree as daemon-tested then writes a marker with "os": "Linux".
    • nightly-next keeps promoting.
  • Failure:
    • A Linux-only red in the gate.
    • A green gate with no marker.
    • nightly-next more than 30 min behind.
    • A shadow red, which is a macOS regression to fix but not a gate.
  • Rollback: revert this PR.

Agent session: agent session

šŸ¤– Generated with Claude Code