ci: skip the delta diff when main moved further than the pull request

Merged
TE
20d ago
+39-3

Summary

#14961 and #14960 failed suite-coverage with coverage_gap=true and about 130 unrelated unit_selectors, though neither touches cmuxUITests/. Both had just merged main into a green head, so the delta step (#14659) routed diff(green head, merge):

  • #14961 (run 36315941692): delta since green head ff7939d5f7: 40 files (pull request diff: 15 files)
  • #14960 (run 36315989569): delta since green head 7ae89a2bfb: 43 files (pull request diff: 10 files)

The delta carries everything main gained since the pull request last synced. Here that included #14966's six cmuxUITests/ edits plus main's cmuxTests/ and app changes, so the pull request was charged with main's work. Reproduced locally: git diff --name-only ff7939d5f7 ef8c94c28f lists 40 files, 6 under cmuxUITests/; git diff --name-only ef8c94c28f^1 ef8c94c28f lists 15, none there.

The fix is in scripts/ci/delta_since_green.py. It now skips the delta, which means routing the usual pull request diff, in two cases:

  • the delta has more files than the pull request diff. The delta is there to route less, and main's own runs cover what main gained.
  • the delta carries a cmuxUITests/ edit that the pull request didn't make. No pull request job runs those, so choose_ci_suite.py would count main's edit as this pull request's coverage gap.

In the last 50 failed pull request runs of ci.yml, these two were the only spurious suite-coverage failures. The other failing runs (five from #14903, one each from #14928, #14883 and #14926) all come from PRs that edit cmuxUITests/.

Testing

  • python3 -m unittest tests/test_ci_delta_since_green.py passes (30 tests). There are two new cases: a delta larger than the pull request diff, and main editing cmuxUITests/. Two existing fixtures now give the pull request two files, so their two-file delta still applies.

🤖 Generated with Claude Code

Summary by cubic

Fixes spurious suite-coverage failures when main moved further than the pull request by skipping the delta diff in those cases. Previously the delta routed main's own changes, including cmuxUITests/ edits, and the coverage check falsely charged the pull request for them; now the script routes the usual pull request diff whenever the delta is larger than the pull request diff or carries cmuxUITests/ edits the pull request didn't make.

Two new test cases cover both skip conditions.