ci(stable-next): fix what the first nights found (Chromium keychain, soak, perf bun, suite watchdog)

Merged
AZ
3d ago
+74-12

The cmux NEXT Stable pipeline (#27898) is live: main's tick dispatched a publish run on nightly-next each night from 2026-10-31. Every run failed, so no stable build has shipped. This fixes the four defects the runs exposed in the pipeline itself.

What changes

  • UI tests on the signed app: the fresh test home has no login keychain. Chromium's first Safe Storage read put up a "Keychain Not Found" system dialog and held its pages, so the Chromium step failed on macOS 26 and 27 (shot-04-chromium.png, run 41419773317). WebKit loaded behind the dialog. A real install has a keychain.
    • The journey now launches the app with CMUX_MOCK_KEYCHAIN=1. CEFSwitches honours it in release builds, and it also turns the password import off, so the throwaway home stores nothing real.
    • fresh_home_launch_smoke.py gains an EXTRA_ENV hook. It is empty for the nightly launch smoke, which is therefore unchanged.
  • UI tests, terminal step: the step compared shown-tab counts. New Tab from Home opens a new workspace with one tab, so the count stays 1. The step now needs the tab the action made to be the selected tab in a shown pane. The echo step right after it was already passing in that same terminal.
  • Soak: it required the build to still be on the cmux NEXT feed. That feed keeps 2 builds, so newer nightlies dropped the picked one before the gates ended (refused: build 4141347814401 is not on the cmux NEXT feed). It now checks pulled.json and the canary verdict only, through stable_next.py soak.
  • perf-budgets: the perf build makes the web bundles and needs bun 1.4.4 (error: the cmux-next web bundles need bun 1.4.4). Added setup-bun, as cmux-next-perf-budgets.yml has.
  • Swift suites: CmuxNextAppTests runs for about 15 minutes on a GUI mini. The lane's 900 s per-suite watchdog cut it off every night (stalled/timeout 908s). The per-suite limit is now 2400 s, and the job limit 240 min.

Not in this PR: the other suite failures on the nightly commits (ClassicUpgradeSimulationTests, WindowCleanupCoverageTests, PaletteRootMenuTests, CEFRendererPathsTests, …). These are tests failing on the shipped code, not pipeline defects, and stable correctly refuses those builds.

Observed / Not run

  • Ran:
    • python3 tests/test_stable_next.py passes 21 tests, including the new soak tests. A build off the feed passes; a pulled or failed build is refused.
    • tests/test_fresh_home_launch_smoke.py (13) passes.
    • An env check confirmed that the nightly smoke's fresh_env has no CMUX_MOCK_KEYCHAIN, while the journey's does.
    • These also pass: actionlint, test_ci_runs_on_fallback, test_ci_macos_xcode_selection, test_ci_fork_runner_routing, test_ci_upload_artifact_retention, test_supplychain_security, test_ci_owned_pool_rescue.
  • Dry run on this head (stable-next-dry-run): the link is in a comment below once it runs.
  • Not run: a real publish, which happens only on nightly-next.

Performance

No perf numbers: CI pipeline only, no app code.

Changelog

none

🤖 Generated with Claude Code

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Fixes the five pipeline defects that made every nightly cmux NEXT stable publish run fail, so stable builds can ship again.

  • Launches the UI test app with CMUX_MOCK_KEYCHAIN=1 so Chromium doesn't block on a missing-keychain dialog in the fresh test home; the nightly launch smoke is unchanged.
  • The terminal step now requires the tab created by New Tab to be selected in a shown pane, instead of comparing shown-tab counts (New Tab from Home opens a new workspace with one tab).
  • The soak gate no longer requires the build to still be on the cmux NEXT feed, which keeps only 2 builds; it checks pulled.json and the canary verdict only, and fails closed on an unreadable pulled.json.
  • The soak check runs again right before the feed write, so a rollback during the gates still stops the publish.
  • Installs bun 1.4.4 via setup-bun for the perf budgets job, which builds the web bundles.
  • Raises the per-suite Swift watchdog from 900 s to 2400 s and the job timeout from 150 to 240 min.

Not covered

  • The remaining nightly suite failures (ClassicUpgradeSimulationTests, WindowCleanupCoverageTests, and others) are tests failing on the shipped code, not pipeline defects.