CI: run the Firefox/Safari/phone tests alongside Chromium, and skip them on main when a draft run already passed the same tree

Merged
BI
1mo ago
+208-74

User description

Requested by Josh · project thread

Summary

Before: the Firefox, Safari and phone browser tests (release-browser-matrix) waited for all three Chromium Production UI shards to finish before starting, although they only need the shared Playwright build. If any Chromium shard failed, the whole cross-browser matrix was skipped. And after every merge, main re-ran the whole matrix, even when the draft's full-suite run had just passed the identical tree.

After: the matrix starts as soon as the shared build is ready, in parallel with Production UI. Desktop Safari (WebKit) and Firefox split four ways instead of three. On a push to main, the matrix is skipped only when a green full-suite workflow_dispatch run on the merged PR head recorded testing the byte-identical tree within 24 hours, with every matrix leg green. Every test still runs somewhere, one worker, zero retries, same timeouts.

Measured (job timings read from GitHub):

  • Main run 38352161182: build ready at minute 4.1, matrix idle until 15.2, run ended 34.1. Slowest legs were WebKit 17.2 min and Firefox 13.4 min of tests.
  • Dispatch runs 38369056741 and 38370421833: one red Chromium shard each, matrix skipped entirely.
  • Expected: full runs about 20 to 23 min instead of 34 to 45; a main push whose draft already passed the full suite should skip the matrix and finish in roughly 15 min.

How:

  • Drop ui-critical from the matrix's needs/if. The Chromium leg's mockups-only path keys on ui_changed, because production Chromium is then proven by Production UI in the same run or by the merged PR's identical tree.
  • WebKit and Firefox leg counts 3 → 4.
  • scripts/ci-main-tree-proof.mjs gains a matrix group, proved only by a dispatch run under the same checks as the existing proof (same repo, same head, its own tested-tree record with event=workflow_dispatch equal to tree(HEAD), fresh, every leg green). The job's if adds tree_proven_matrix != 'true' inside its main-push clause only, so schedule, release branches and dispatch runs are unchanged.

Areas touched: Delivery and assurance

Verification

  • node scripts/ci-main-tree-proof.mjs --self-test: ci-main-tree-proof self-test passed. (new cases: red leg, missing legs, other tree, wrong event record, red or stale dispatch run, tree mismatch, dispatch events never skip their own matrix)
  • Every test file that reads ci.yml or the proof script: 690 passed, 2 failures in tests/test-runner-safety.test.ts (heavyweight lock) that fail identically on unmodified main in this container.
  • ESLint and Prettier on the changed files: clean.

Risk and rollout

  • Risk: more jobs run at once; on a red Chromium run the matrix spends runner minutes instead of being skipped. The matrix proof is fail-closed: any doubt runs the matrix on main as before.
  • Rollback: revert the PR.
  • Provider or production effects: None.

Notes

Not done: refresh the release shard timing tables (they under-predict WebKit by about 25%). That needs the release-ui-timings artifacts, which this container cannot download.

🤖 Generated with Claude Code

https://claude.ai/code/session_014qMDuRLipTaGrY6EyC89hm

CodeAnt-AI Description

Run cross-browser tests sooner and skip verified repeats on main

What Changed

  • Firefox and Safari tests now start alongside Chromium instead of waiting for its test shards, so cross-browser failures are reported in the same run even when a Chromium shard fails.
  • Firefox and desktop Safari tests split across four runners, shortening their test legs while keeping all tests.
  • Main skips the release browser matrix only when a recent successful full-suite run tested the identical tree; scheduled and manually dispatched runs still execute the matrix.

Impact

✅ Earlier Firefox and Safari test results
✅ Cross-browser failures reported with Chromium failures
✅ Shorter main-branch CI after a verified full-suite run

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.