Cut CI wait: split the tablet and Safari release legs, rebalance PR browser groups, stop a frozen unit batch sooner

Merged
BI
1mo ago
+151-56

User description

Requested by Josh · project thread

Summary

Before: every merge to main took 36 to 40 minutes of CI before Railway would deploy, because "Wait for CI" waits for the slowest job, and the tablet browser leg ran the whole tablet suite on one runner. On main run 38053471704 it hit the 30-minute global timeout with 345 tablet tests never run. A ready PR took about 14 minutes, set by one unbalanced Production UI shard (521s of tests against 401s and 346s). A frozen unit coverage partition on #3450 held its run to the 20-minute job cap.

After: tablet splits into four legs and desktop Safari into three, so every tablet test now runs and the slowest release leg drops from ~35 to ~13 to 17 minutes. The three Production UI shards are rebalanced to 422/426/420s, about 1.5 minutes off every ready PR. The unit test step stops at 13 minutes instead of 20 when a batch freezes.

How: the tablet timing table comes from the per-test durations in that tablet leg's log; files the leg never reached carry the mobile-webkit cost and are marked as estimates until a tablet release-ui-timings report replaces them. Production UI profiles are re-measured from the three shard logs of PR run 38101082169, with four files moved. Every test still runs once, on one worker, zero retries, same per-test timeouts.

Areas touched: Delivery and assurance

Verification

  • Checks run: npx vitest run tests/playwright-pr-shards.test.ts tests/playwright-release-shards.test.ts tests/ci-*.test.ts → Tests 247 passed (247); check:diff-integrity PASS (17 -> 17 test cases); check:github-actions and check:ci-scope passed; node scripts/playwright-pr-shards.mjs --validate → PR UI shard parity OK: 46 production specs across 3 groups. Full suite left to CI.

Risk and rollout

  • Risk: low. Workflow and shard-grouping only. Main pushes book 4 more runners for the report-only browser matrix (21 legs, was 17). Tablet estimates could leave a leg lighter or heavier than planned; each leg is far under the 30-minute cap either way.
  • Rollback: revert this commit.
  • Provider or production effects: None. The main-push saving appears once a merge runs the new matrix.

Notes

  • Non-UI merges (where UI was never proved on the PR) still run the full Chromium suite on one release leg (~28 min); splitting that is a possible follow-up.
  • Many PR runs are cancelled by bot "fix:" commits (Qodo pushed three in 20 seconds on #3450), each restarting the clock.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JoxhPS6iT7SYdBZeSR1w1S

Generated by Claude Code

Summary by cubic

Cuts CI wall-clock time on merges and PRs by splitting the slowest browser legs, rebalancing the PR UI shards, and failing a frozen unit batch sooner.

The tablet release leg ran the whole suite on one runner and hit the 30-minute global timeout with 345 tests never run, holding Railway's "Wait for CI" for ~35 minutes on every merge. It now splits into four legs (~12 minutes each) and desktop Safari into three (~13 minutes), so every tablet test runs and main finishes in roughly 13–17 minutes instead of 36–40.

The three Production UI shards drifted to 521s/401s/346s of tests, with the heaviest setting every PR's wall clock. Re-measured timings and four file moves bring them to 422/426/420s, trimming ~1.5 minutes off each ready PR.

The unit test batch with partial coverage now fails at 13 minutes instead of 20 when a partition freezes; tests already have 30-second timeouts, and nothing is skipped or retried.

Migration

  • Main pushes now book four more runners for the release browser matrix (21 legs, was 17).
  • Tablet timings for files the timed-out leg never reached are marked as estimates from mobile-webkit until a tablet release-ui-timings report replaces them.

CodeAnt-AI Description

Shorten CI waits and ensure tablet browser tests complete

What Changed

  • Tablet browser tests now run across four jobs, and desktop Safari tests across three, so tablet checks can finish instead of timing out with tests left unrun.
  • Production UI checks are redistributed to reduce the slowest PR test group by about 1.5 minutes.
  • Frozen unit test runs now stop after 13 minutes instead of waiting for the 20-minute job limit.

Impact

✅ Shorter release waits
✅ Complete tablet browser test runs
✅ Earlier failure for frozen unit tests

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.