Swift tests: name the suite that leaves windows open, and a .closesItsWindows check that closes them and fails that suite

Merged
TE
8d ago
+471-0

Leo, on Swift suites that leave windows open for the suites after them: "But why do you guys not just clean up after yourselves, or at least do your best to?"

What this adds

The broad Swift pass grew from 3 to 428 open windows in one test process, and since every Core Animation commit lays out each open window that needs it, throughput fell from about 2,000 tests a minute to about 100 and the pass was cut short. This PR is the shared hook and the check; the suite fixes follow in their own PRs.

  1. Every window is tagged with the suite and test that opened it. Tests/Support/CmuxNextTestExitTrace/CmuxNextTestWindowCensus.c (a new file in the existing test-only C target; the app links none of it) swaps NSWindow's two designated inits to record swift-testing's Test.current (as Module.Suite and the test), and -close / -orderWindow:relativeTo: to record whether the window was closed. Windows from suites running in parallel are told apart by their tag, not by timing.
  2. A census at the test process's exit names, per suite, the windows still open: cmux-next window leak: <Module.Suite> left N of the M windows it created open (K on screen): <class> "<title>", ..., after one summary line. It needs no change to any suite, so a suite that opens windows without the check below is still named.
  3. @Suite(.closesItsWindows) (CmuxNextTestSupport/WindowLeakCheck.swift): when the suite ends, it closes every window its tests opened and left open (isReleasedWhenClosed = false, then close()), logs them on stderr, and records an issue on that suite, naming the suite, each window's class and title, and the test that opened it. It is a normal test failure on the suite that leaked, not a crash and not a later suite's failure. A window its test closed is not a leak, even while something still holds it. One main-actor turn passes before a window counts, so a window released without a close is not reported when its autorelease pool drains.

The check fails only suites that carry the trait, so it cannot turn base red on its own. Suites get the trait in the follow-up PRs together with their own teardown fixes; the last of them adds a test that every suite in a file that opens a window carries it.

Tests

WindowLeakCheckTests (CmuxNextAppTests): a window and a panel are tagged with this suite and test; the check closes and names a window the suite left open; a closed window that is still held is not a leak; ordering a closed window back in reopens it.

Measurement

Same harness at this head (the trait is on no suite yet, so this is today's leaking): scripts/ci/package-test-lane.sh suite Packages/macOS/CmuxNext through the fleet, one test process per filter (CmuxNextAppTests by first letter, every other target alone), 32 processes that reached their end.

metricbefore (this head, no suite fixed)afterdeltaharness
windows still open at process exit, summed251 (42 on screen)follow-up PRsfleet steps 173f76a44862a5c04019cbd6, 9f91aca811bb76a732202fef, 448e6672289e25f8f20c6316
suites named as leaking79 suites, 233 windows (18 more opened outside any test)follow-up PRssame
closed windows something still holds1,027same

Worst first: SidebarRevealSelectionTests 12, SidebarPinnedRowWidthTests 12, SidebarRowsMoveOnRenderServerTests 9, MultiWindowInputRoutingTests 9, SidebarRowHoverInsetTests 8, SidebarMenuAnchorTests 8, PageFullscreenWholeScreenTests 8, BrowserToolbarPressTargetTests 8, PaletteNavigationTests 7 (all 7 on screen), UpdateReadyCardLayoutTests 7. The usual cause: a test's teardown calls orderOut(nil), which hides the window but keeps it in the app's window list, or the test never closes it.

Two processes gave no census: CmuxNextAppTests.C (the Cloud broker hang #26809 tracks) and CmuxNextAppTests.G (signal 11; the same filter crashed with signal 11 at the base commit in fleet step a0898dde41e141132554161e, and passed at this head in step 6afe330519763d2a4bd29aca).

Risk: test harness only. The swaps only add a tag and a flag around AppKit's own implementations, in the test bundle; no product code changes.

Agent session: agent session

šŸ¤– Generated with Claude Code