ci: place the Iroh release gate's Tailscale job on the owned minis

Merged
TE
19d ago
+222-10

What

The Iroh release gate's two macOS jobs always took Blacksmith macOS 15, even while owned minis had free runners. This routes Tailscale version-skew compatibility to an owned Mac through a picker; simulator-e2e stays on Blacksmith (reasons below).

How

  • iroh-release-gate.yml gets a Linux runner job, like test-e2e.yml's: on attempt 1 of a trusted ref (resolve-dispatch-ref.yml's trusted_ref) with CI_PR_POOL_OWNED == '1', it runs scripts/ci/e2e_runner_pool.py (live owned runners through the routing App token when it mints) with no queue rounds, so it names an owned label only while a machine is free now. It outputs label only for a glaeda-* pick; any other answer, a skipped pick or a failure is empty.
  • tailscale-version-skew runs-on: github.run_attempt == 1 && needs.runner.outputs.label || <the old Blacksmith macOS 15 expression>. It runs if: !cancelled() && needs.resolve-ref.result == 'success', so a failed runner job still falls back to Blacksmith. On an owned pick CMUX_CI_XCODE_APP follows CMUX_CI_XCODE_APP_PR (the minis carry only that Xcode). Its EXEMPT entry in the Xcode-pin guard is gone, since the pin now follows the lane.
  • A new step takes glaeda's gui token (glaeda-canonical-root take-gui, as in app-host-unit-tests and test-e2e's test) before the gate's app-host tests. It is a no-op on Blacksmith.
  • The runner job uploads the same macos-pool-persistent-...-1-<label> and owned-pool-watch markers. queue_janitor.OWNED_DISPATCH_WORKFLOWS and owned_pool_rescue.DISPATCH_WORKFLOW_PATHS now include the workflow. A stuck or refused owned job is re-run as attempt 2, which takes Blacksmith.
  • Tests: IrohReleaseGateWiring (runs-on, Xcode pin, fallback, trust gate, janitor), plus a rescue target case. docs/ci-runners.md table rows.

Constraints checked

Why macOS 15? No job needs it. The Blacksmith macOS 15 image with CMUX_CI_XCODE_APP_MACOS_15 (Xcode 26.3) was the default lane when the workflow was written. The Tailscale job already sets CMUX_CI_REQUIRED_MACOS_SDK_MAJOR: 26, so it wants an SDK 26 Xcode, which the minis have (the lane pin, 26.6). It needs no iOS simulator, no network privileges and no Tailscale install: run-iroh-tailscale-compatibility-gate.sh runs 4 cmuxTests/IrohTailscaleVersionSkewMacGateTests app-host tests (DerivedData in $RUNNER_TEMP) and 9 filtered swift tests from three packages (scratch in $RUNNER_TEMP). It uses no secrets. The checkout keeps persist-credentials: false.

simulator-e2e does not move. It writes the staging account, or the production Stack server key, into $HOME. It builds into the shared ~/Library/Developer/Xcode/DerivedData, writes the tagged app's defaults and keychain items, launches the Mac app and boots a simulator in the console session, and runs for up to 2 hours. docs/ci-runners.md keeps jobs with secrets off the owned Macs. glaeda's hook keeps ios-streamed-validate's validate, which has the same shape, on Blacksmith. A glaeda-ios-sim pool exists (8 machines), but the secrets and shared-state reasons still apply.

glaeda hook class. Neither tailscale-version-skew nor simulator-e2e is in JOB_CLASSES or WORKFLOW_JOB_CLASSES on glaeda main. An unknown cmux job is admitted as compile, pinned to root 1: it gets 2 units plus the persistent-DerivedData and root-1 tokens, and no gui token. test-e2e used the same arrangement before it had a class ("glaeda gives an E2E job, which it does not know, the mini's root token"). So the picker's root label (glaeda-root-<class>-xcode-<version>) is the right one. That class is safe but broader than needed, because the job never touches the canonical root. The take-gui step covers the one gap, the shared console session and testmanagerd. Follow-up for glaeda: class ("iroh-release-gate.yml", "tailscale-version-skew") as gui (on a two-root mini that is the gui token and 1 unit, no root). The take-gui step then returns 0 at once.

Label. The picker returns the root label of the first owned pool with a free machine (std, then light), because the job holds a root token. It is not the side label (side runners never hold root tokens) and not glaeda-ios-sim (no simulator).

Rescue trade-off. simulator-e2e runs in the same run, so a rescue can affect it. If the owned Tailscale job is stuck, the rescue cancels the run, simulator-e2e included, and re-runs the failed and cancelled jobs on Blacksmith. If the job is refused, the rescue waits for the run to finish, up to the 150-minute watch. The picker takes an owned Mac only while one is idle, so stuck jobs should be rare.

Proof

Dispatch on this branch (mode=private-path, staging): RUN_PROOF

🤖 Generated with Claude Code

Summary by cubic

Routes the Iroh release gate's Tailscale version-skew job to the owned Macs when one is free now, keeping Blacksmith macOS 15 as fallback.

  • A new Linux runner job (using e2e_runner_pool.py) names an owned Mac only on attempt 1 of a trusted ref with no queue rounds; any failure, empty pick, or re-run falls back to the old Blacksmith macOS 15 expression.
  • On an owned pick the job switches to the lane's Xcode pin and takes the mini's gui token for its app-host tests; Blacksmith keeps the old pin and that step is a no-op.
  • The owned-pool rescue sweeper and queue janitor now know the workflow, so a stuck or refused owned job is re-run as attempt 2 on Blacksmith.
  • simulator-e2e stays on Blacksmith because it writes staging/production secrets to $HOME and mutates shared user state.

Summary by CodeRabbit

  • CI Improvements
    • The Tailscale compatibility check can run on an available owned Mac for trusted, first-attempt runs; otherwise, it uses the existing macOS runner fallback.
    • The check now obtains a GUI session token when available, with defined handling for expected outcomes.
    • If an owned runner is unavailable or a run becomes stuck, recovery can rerun eligible jobs on the fallback runner while preserving jobs that passed.
    • Runner eligibility and recovery behavior are now covered by automated checks and CI documentation.