ci: webview flakes on the base only warn; split accent-glass; name new webview reds

Merged
TE
6d ago
+967-182

Leo's ask, relayed: "Webview tests keep breaking base as fast as they get fixed. Leo wants them caught at PR time, as a system fix rather than more red-fix PRs."

Of the 22 PRs that brought a webview red to feat-cmux-next since 2026-10-28, 14 merged with react-apps-base-check finished and red, and 8 merged while it was still pending. Per-diff routing never skipped the suite. A blocking verdict can't be built on what the gates read today, for three reasons:

  1. Flakes count as new reds. The base replay runs each red file once, on the merge parent. A test that fails now and then on the base can pass that one run by chance, and the red is then charged to the change. agent-pane-suggested-tasks was charged on 9 of 40 sampled red PR runs, and newtab-project-chip-fit the same way.
  2. accent-glass-invariants timed out. It scans 2,542 stages in about 9 minutes alone, against a 900 s limit. On a busy shard it timed out with no room left before the retry deadline, so it was charged as new (runs 41079754472, 41076022470, 41048809560).
  3. The red-checks comment called the wrapper checks "base red." That held even when a job they gather was the change's own red, and a red react-apps-base-check read "no finished base run to compare."

This PR fixes all three. It changes no workflow permissions and posts no new statuses. A follow-up gate (merge_guard / next-autoland) reads the existing web / react-apps-base-check (N) conclusion and log.

Changes

  • scripts/ci/webviews_test_history.py (new):
    • record adds one push run's per-test webview verdicts (from the shards' results) to a history of the last 120 runs, with the run's inputs key.
    • The inputs key is the git ids of every path web_subareas.py routes the webview suite on.
    • flaky names a test only when two base pushes with the same inputs gave it different verdicts. Most pushes change no webview input, so that evidence builds up fast. A break and its revert (green, red, green) change the inputs each time, so they are never flaky, and a re-run of a push counts as more evidence.
    • A file that timed out says nothing about its tests on that run.
    • web_subareas.py now routes the suite on for this script too.
  • ci-web.yml:
    • On pushes, each shard uploads its results.
    • A new push-only upkeep job, webviews-test-history, saves the history to the Actions cache under webviews-test-history-<branch>-<run>. Every step in it is continue-on-error, like webviews-test-durations.
    • react-apps-base-check restores the base branch's history with read-only actions/cache/restore and passes the flaky names to the comparison. PR runs never save it, so a PR can't write its base's history.
    • The base check now runs the head's copy of guards_new_reds.py and of the history script, which the head shard publishes beside its runner, as it already does for run-webviews-tests.sh.
  • guards_new_reds.py --flaky FILE: a listed red that the base replay passed becomes ::warning title=Webview test flaky on the base:: and the step stays green. Every other new red is still ::error ... red on this change:: and fails the step.
  • accent-glass-invariants is split into four parts (accent-glass-invariants-1..4.test.ts, 480 s each). Each part scans every fourth stage of the sorted list. The body moved unchanged to test/support/accentGlassInvariants.ts. The stale-baseline warning covers only the part's own stages.
  • pr_red_checks_comment.py:
    • web / react-apps-check and web / Web status are never called base reds. Each follows only the jobs it gathers: react-apps-check follows its shards, the base replays and react-doctor; Web status follows the other web / jobs.
    • An aggregate blocks when a job it gathers blocks, when one of them was cancelled, or when none of them is red. It is excused only when every red job it gathers is a base red. A red it doesn't gather, such as the size cap, never excuses it.
    • A red react-apps-base-check (N) names up to 3 of its new red tests, read from its "Webview test red on this change" annotations, and blocks.

Repo rule note: CLAUDE.md asks not to add CI jobs or steps. The new job and steps are upkeep, never a gate, and nothing in them can fail a run. Leo's standing orders make CI changes the lanes' call after independent review.

Until this lands, base-check PR runs use the base's scripts, so the flake handling takes effect on PRs opened after it merges. The history fills from the first feat-cmux-next pushes after that, and a flake needs two runs of the test on the same inputs.

Observed / Not run

  • Codex review (gpt-6-astra) BLOCKed twice, and both are fixed:
    • stage names with spaces in the accent-glass baseline;
    • break-and-revert treated as a flake, and aggregates excused by unrelated base reds.
  • Ran here: tests/test_webviews_test_history.py (16 OK, new), tests/test_guards_new_reds.py (OK, with 3 new --flaky cases), tests/test_webviews_tests_new_reds.py (OK; the workflow-step tests now publish the two scripts with the runner), tests/test_pr_red_checks_comment.py (27 PASS, 7 new), tests/test_cmux_next_web_base_push.py, tests/test_webviews_tests_per_file.py, test_runner_label_policy, test_ci_runs_on_values, test_ci_runs_on_fallback, test_runs_on_labels, test_ci_blacksmith_linux_failover, test_ci_pr_runner_pool, test_ci_change_areas and test_test_reds (all OK).
  • tests/test_ci_fork_runner_routing.py fails with the same single failure on the base (cmux-next.yml:1304 CI_SESSION_LANE_RUNNER), so it isn't this change's.
  • The last commit wires tests/test_webviews_test_history.py into cmux-next.yml's guard list (the every-guard-is-wired check) and gives the history upload step an id the gate reads as a warning (the continue-on-error id check).
  • CI at 31a587e6: every check green, including the four react-apps-check shards (they run the four accent-glass parts) and all four react-apps-base-check replays. Codex review (gpt-6-astra): SHIP at d5ca5b48, then SHIP on the delta to 31a587e6.

risk: normal (CI scripts and tests; no product code).

Changelog

none

Agent session: agent session

šŸ¤– Generated with Claude Code