ci(webviews): every timed-out file gets its alone retry before it counts as red
Red fix for web / react-apps-check: a file that times out in a contended batch now gets its alone retry before it counts as red.
What was wrong
React-apps-check shards run plain files 32 at a time on the 32 vCPU runner. About 10 browser files time out at 120 s together. run-webviews-tests.sh gave a timed-out file its alone retry only when its whole limit still fit in CMUX_WEB_TEST_TIMEOUT_RETRY_BUDGET, which was a fixed 240 s. That fits 2 or 3 retries. The rest kept TIMEOUT, for example in job 123436110075 (#27007's shard 1): 3 got retries and all passed, and 7 kept TIMEOUT. react-apps-base-check then replays those files one at a time, they pass, and guards_new_reds.py reports them as new reds on every web PR. A timeout is never a base red, so lanes can't excuse them.
The change
scripts/ci/run-webviews-tests.sh: withCMUX_WEB_TEST_TIMEOUT_RETRY_BUDGETunset (the default), every timed-out file gets one retry, alone, at its full limit. A newCMUX_WEB_TEST_RETRY_DEADLINE(seconds since the runner started) bounds all retries. A timeout retry starts only if its whole limit fits before the deadline. A failed file's retry is cut at the deadline and keeps its first named failure, like the existing shared-budget cutoff. An explicitly set timeout budget still works as before..github/workflows/ci-web.yml:CMUX_WEB_TEST_RETRY_DEADLINE: "1920"on the shard step, and the failed-file budgetCMUX_WEB_TEST_RETRY_BUDGETgoes from 300 to 1920 so the deadline, not the budget, bounds those retries too. This PR's own shard 4 (run 40881495552) showed the same defect for failed files: the 300 s budget cutaccent-glass-invariants' alone retry (it needs more than 297 s) and skippedtranscript-thoughtandstreaming-text-frames, and the base replay counted all three as new reds.tests/test_webviews_tests_new_reds.pynow pins budget >= deadline <= 34 min instead of the literal 300. The step starts about 2 min into the job, so retries end by about 34 min, inside the 40 mintimeout-minutes, with room left for the uploads.- Retries still hide nothing. A file that times out again alone stays TIMEOUT, and a file that fails alone stays FAIL with both logs. The base replay runs
CMUX_WEB_TEST_RETRIES=0, so it is unchanged. No test, limit, allowlist or red rule changes. Browser concurrency is unchanged (#27394 handles that separately; this PR is the retry backstop and doesn't depend on it). - Tests (
tests/test_webviews_tests_per_file.py): every timed-out file is retried by default, including when their limits sum past the old 240 s, and a file that fails alone stays red with its named failure. A retry that runs almost to its limit doesn't cost the next file its retry (the independent review's round-1 finding). A retry that doesn't fit before the deadline keeps TIMEOUT or FAIL. Both of the first two tests fail against the old script (run 40877149558). All 44 runner tests andrun-webviews-tests-retry.test.shpass at the head (run 40879880405).
Proof: CI shard 1's file batch, 32 at a time, base tip 400cb089
Same harness both times: cmux-ci test --setup webviews on Blacksmith 32 vCPU, with the 71 files of job 123436110075's shard 1, plus one planted file that always fails and one that always hangs. Settings are CMUX_WEB_TEST_JOBS=32 and CMUX_WEB_TEST_RETRY_BUDGET=300, as in CI.
| metric | before (old script) | after (round-1 head) | after (final head 0ec9d9ca) | delta (before to final) | harness or run link |
|---|---|---|---|---|---|
| files timed out on first run (incl. planted hang) | 12 | 11 | 11 | -1 (load noise) | before 40876865326, round-1 40876861671, final 40879917068 |
| timed-out files retried alone | 2 | 11 | 11 | +9 | same |
| timed-out files left TIMEOUT without a retry | 10 | 0 | 0 | -10 | same |
| files that passed alone after a first-run TIMEOUT | 1 | 9 | 8 | +7 | same |
| reds at the end | 16 | 7 | 8 | -8 | same |
| planted always-fail / always-hang | FAIL / TIMEOUT | FAIL / TIMEOUT | FAIL / TIMEOUT | unchanged | same |
Before: timeout retry budget (96s left) cannot fit test/agent-pane-dock-gutters.test.ts's 120s limit: keeping TIMEOUT repeats for 10 files. After: every one gets RETRY ... after its timeout. Nine pass alone (agent-pane-short-pane-composer, agent-pane-dock-gutters, transcript-parked-stream, agent-pane-wheel-pass-through, chat-inspector-push-motion, thread-minimap-left-rail-stays, agent-pane-scrollbar-edge, agent-pane-running-layout, chat-environment-scroll-stability). transcript-gap-select-browser fails alone. It also fails alone on the base replay (jobs 123430316969, 123430796812, 123436108553), so it is now a named FAIL that base shares, where before it was a TIMEOUT that couldn't be excused. The remaining reds (chiefLook, rowSummary, glyph-alignment, transcript-fold-no-blank-space-browser) are failed files, the same before and after. In the final-head run, thread-minimap-left-rail-stays also failed alone, so it stays red as a named FAIL that the base replay can compare (it had passed alone in the round-1 run). Retry wall time: about 6.5 min for 17 retries.
Risk
Low. CI-only, in the per-file webviews runner. Worst case, a shard spends up to the deadline on retries, and that stays inside its 40 min job. Independent review: Codex gpt-6-astra high via review-router. Round 1 was BLOCK (the summed default budget could still skip the last retry), which is fixed here. Round 2 was SHIP at 0ec9d9ca. Round 3 (the failed-file budget and the pin test) was SHIP at 7ebbc6e24ae994dfc5d9a6f4afd1407bb177810f.
Agent session: agent session
š¤ Generated with Claude Code