ci(stable-next): a real login keychain for the UI tests; suites back on the fleet; node for perf

Merged
AZ
2d ago
+66-24

Night 2026-11-03 (run 41582798853) was the first stable run with #32978's fixes, and most of it went green:

  • Green: real Developer ID sign, notarize, crash-class and soak.
  • UI tests: 11 of 12 steps passed on both macOS 26 and 27.

Three pipeline defects remained. This fixes them.

What changes

  • Chromium in the UI tests:
    • #32978's mock keychain never applied: the app strips every inherited CMUX* variable at launch (LaunchIdentity.inheritedKeys; its log says cmux: ignored inherited CMUX_MOCK_KEYCHAIN). So Chromium still sat on "Keychain Not Found".
    • Widening that strip list would loosen the launch-identity boundary, so the test home gets a real login keychain instead.
    • stable_next_ui_journey.py creates it before launch through a new PREPARE_HOME hook in fresh_home_launch_smoke.py, which stays None for the nightly smoke.
    • Every security call runs with the fresh home's own HOME and CFFIXED_USER_HOME.
    • Checked over SSH on cmux11s: the keychain becomes that home's default and can be written and read back, and the runner account's real default keychain and search list are unchanged.
    • This also means the gate exercises Chromium's real Safe Storage path, which the mock keychain would have skipped.
    • The unused EXTRA_ENV hook is removed.
  • Swift suites:
    • On a GUI runner, each suite's Swift tests passed, then its XCTest pass hung about 10 minutes on the runner account's Contacts store (Error obtaining permanent ID … incompatible from AddressBook-v22.abcddb).
    • Every suite reported "failed" at about 659 s, and the 240-minute job limit ended the run.
    • The shards go back to build-fleet steps (package-test-lane.sh suite, macos-27 + ram48, never an M6, CMUX_SWIFT_PACKAGE_TEST_TIMEOUT_SECONDS=2400, allowed by the controller up to 3600).
    • They are submitted from the GUI-pool runner, which is a fleet client. The gateway label refused them earlier.
    • The earlier manual fleet run of the same suites gave real results with no such hang.
  • perf-budgets: the webviews app build also needs node 22.18+ (error: the webviews app build needs node 22.18 or newer). Added setup-node 24.22.1, matching cmux-next.yml.

Observed / Not run

  • Ran, all passing:
    • python3 tests/test_stable_next.py (21) and tests/test_fresh_home_launch_smoke.py (13).
    • actionlint.
    • test_ci_macos_xcode_selection (swift-suites exempt again: it only submits fleet steps), test_ci_cmux_ci_client_guarded, test_ci_runs_on_fallback, test_supplychain_security, test_ci_fork_runner_routing, test_ci_upload_artifact_retention, test_ci_owned_pool_rescue, test_ci_nightly_off_blacksmith.
    • The keychain recipe itself, on cmux11s over SSH.
  • Not run: a full UI run with the keychain fix. The next nightly stable run, or a dry run, is the first one.

Performance

No perf numbers: CI pipeline only, no app code.

Changelog

none

🤖 Generated with Claude Code

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Fixes three pipeline defects from the 2026-11-03 stable run: UI tests get a working login keychain, Swift suites return to build-fleet steps, and the perf gate gets Node 24.

  • UI tests: the app strips inherited CMUX* variables at launch, so the mock keychain never applied and Chromium stayed on "Keychain Not Found". The fresh test home now gets a real login keychain via a new PREPARE_HOME hook, exercised with the home's own HOME and CFFIXED_USER_HOME, so the runner account's keychain settings stay untouched and Chromium's real Safe Storage path is covered.
  • Swift suites: each suite's XCTest pass hung ~10 min on the runner account's Contacts store when run on a GUI runner, failing every suite at ~659 s. The shards run as build-fleet steps again, submitted from the GUI-pool runner (a fleet client) with macos-27 + ram48 and 2400 s per suite. A shard still queued after 60 min (exit 75) keeps its place and is never resubmitted; only a finished red or a timeout retries, and the job limit is now 330 min to fit both attempts.
  • perf-budgets: the webviews app build requires node 22.18+; added setup-node 24.22.1, matching cmux-next.yml.
  • Tests: test_ci_macos_xcode_selection records that stable-next swift-suites only submit fleet steps and need no Xcode selection.