ci: fail browser install when system libraries are missing (WebKit could not launch)

Merged
BI
1mo ago
+77-3

User description

Requested by Josh · project thread

Summary

Before: on main, many WebKit legs of the advisory release browser matrix failed every test at browserType.launch because libevent-2.1.so.7 was missing (runs 38224928367 and 38228339791, e.g. release-browser-matrix (webkit, tablet, 4, 4): 196 failed, all at launch). The "Install Playwright browsers" step still reported success.

After: WebKit installs its system libraries and launches on every leg. In dispatch run 38236585559 (head 7e2d712) every WebKit leg's install step passed, including the launch probe, and the log shows libevent-2.1-7t64 being installed.

Cause, confirmed from the logs:

  • The runner resolves apt sources through /etc/apt/apt-mirrors.txt, which points at the azure mirror that stalls. The existing mirror rewrite missed that file, so install-deps ran past the 180 s attempt timeout.
  • timeout killed the attempt shell but not the root apt-get, which kept the apt lock, so retries failed too.
  • On a browser-cache hit the install function had no set -e, so the failed install-deps was masked by the successful cached browser install and the step exited 0 with libraries missing.

How:

  • The mirror rewrite also covers /etc/apt/apt-mirrors.txt.
  • set -e in the release matrix install function and in the same pattern in setup-ui-e2e and setup-lighthouse-chromium.
  • A launch probe of the installed engine runs inside each attempt.
  • Between retries, orphaned apt-get is stopped and sudo dpkg --configure -a finishes any interrupted install.
  • Contract test added in tests/ci-browser-matrix-coverage.test.ts. No test was skipped or disabled; the CI speed-up from #3451 is untouched.
  • A ledger request (docs/outstanding-issues-inbox/) records the genuine test failures below for follow-up.

Areas touched: Delivery and assurance

Verification

  • Checks run: npx vitest run tests/ci-browser-matrix-coverage.test.ts tests/ci-cache-safety.test.ts → Tests 90 passed (90); prettier clean; check:outstanding-issues passed.
  • Hosted proof: browser-matrix dispatch run 38236585559 on this branch. All WebKit, Firefox and Chromium-mockups legs installed and launched.

Genuine test failures now visible (not fixed here)

Now that WebKit launches, 26 tests fail on their own merits (about 1,400 pass). They were hidden before because every test died at launch. Many tablet failures are 1.0 min timeouts and may share a cause.

  • tablet 2/4 (16): ui-accessibility 176, 377, 597; ui-overlap 239, 437, 549; ui-teaching 83; ui-tools 587, 672, 1084, 2183, 2295, 2490, 2674, 2946; ui-universal-search 360.
  • tablet 4/4 (4): ui-my-day-calendar 116; ui-roster-team 100, 174; ui-stress 346.
  • tablet 3/4: ui-cme-phone 351.
  • mobile-webkit 2/5: ui-accessibility 295.
  • mobile-webkit 5/5 and mobile-pwa-standalone 4/5: ui-on-call-boards 558.
  • mobile-pwa-standalone 5/5: ui-phone-scroll-assessments 19; ui-phone-scroll-page-owned 454.
  • tablet 1/4 was still running at the time of writing.

Risk and rollout

  • Risk: low, CI-only. The matrix stays advisory; its WebKit legs will now report the failures above instead of launch errors.
  • Rollback: revert this PR.
  • Provider or production effects: None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCghuougzFFXRF3dQQEcYr

CodeAnt-AI Description

Make browser setup detect missing dependencies and recover from stalled installs

What Changed

  • Browser setup now checks that the selected browser launches; missing system libraries fail during setup instead of causing every test to fail at launch.
  • Setup retries now use the working package mirror and clear stuck package installs before retrying.

Impact

✅ Fewer WebKit tests fail at browser launch
✅ Missing browser dependencies are caught during setup
✅ Fewer retries blocked by a stuck package install

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.