ci: fold obsolete focused gates into truthful app-host shards

Closed
TE
23d ago
+134-289

Summary

The broad app-host classifier now rejects ordinary XCTest assertion failures, Swift Testing failures, earlier-failure/later-clean-summary masking, and any crash/restart/incomplete run. Many focused gates still exist solely to compensate for the old #7471 failure-accounting bug.

Fold those suites back into the deterministic timing-weighted broad shards, while keeping every focused group with a real isolation requirement.

Folded into broad shards

  • main-window zoom placement
  • bundled command PATH
  • agent chat transcript lifecycle
  • agent resume binding
  • browser viewport runtime
  • PTY spawn starvation
  • notification routing
  • browser system proxy
  • Cloud notification dismiss parity
  • agent restore admission
  • Option/Alt modifier
  • ssh-session-attach anchor
  • notification scroll restoration
  • Cloud workspace rename parity

These steps owned 26 full suites. They are removed from FOCUSED_GATE_SELECTORS, so the existing sharder assigns them exactly once by measured suite cost.

Still isolated

My Devices, renderer-memory measurement, Pi Feed global state, destructive remote-tmux topology, focused WebKit portal tests, remote-tmux identity/active-pane gates, socket ACL reload, settings persistence, workspace-switch layout fault, terminal portal visibility, numeric locale, global search, Cloud machine ordering, and notification semantics.

Those have process/global-state, destructive-topology, positive-execution, memory-measurement, or dedicated-host reasons independent of failure accounting.

Measurement

Current-head parent run 35632632906:

  • shard 5 obsolete-accounting steps before the retained failure consumed 79s for the first five gates; measured notification-scroll + rename add ~28s in recent full runs, about 107s focused wall
  • shard 6 obsolete-accounting steps consume 142s focused wall on the current run
  • the split-theme package gate removed in #13423 is another 28s on this run
  • focused xcodebuild startup is typically 10–17s before the first test starts; this PR removes 18 dedicated app-host launches
  • retained focused wall observed on current shard 6 before broad batches is 145s; notification semantics adds ~31s test execution plus five fresh-host launches, so the reservation becomes 235s
  • shard 5 reservation becomes 138s, matching ~124s of retained observed focused work plus headroom

Reservations:

  • shard 5: 245s → 138s
  • shard 6: 264s → 235s (after #13423 already removes the split-theme duplicate)

Coverage and failure accounting

tests/test_ci_app_host_test_output.py already proves:

  • ordinary 1 failure (0 unexpected) is red
  • an earlier failure cannot be hidden by a later clean summary
  • Swift Testing failures are red
  • restart/timeout followed by a passing subset is incomplete and red

This PR adds a sharding guard listing all 26 folded suites and proving each is discoverable in broad packing with no remaining focused owner. Existing sharding tests prove all discovered selectors are assigned exactly once and retained focused suites stay excluded.

Known process-global/crash/topology suites remain isolated.

Stacked on #13423 → #13386.

Related: #6134, #7471, #12180, #13095, #13325.

Summary by cubic

The broad app-host shards now fail on ordinary assertion failures, Swift Testing failures, earlier-failure/later-clean-summary masking, and crash/restart runs, so the focused gates that only compensated for the old #7471 failure-accounting bug are obsolete. Removes 18 dedicated app-host launches and folds 26 suites back into the timing-weighted broad shards, cutting shard 5's reservation from 245s to 138s and shard 6's from 264s to 235s.

Still isolated

  • Suites remain focused when they need process/global-state, destructive-topology, positive-execution, memory-measurement, or dedicated-host reasons: My Devices, renderer-memory measurement, Pi Feed global state, destructive remote-tmux topology, WebKit portal tests, remote-tmux identity/active-pane gates, socket ACL reload, settings persistence, workspace-switch layout fault, terminal portal visibility, numeric locale, global search, Cloud machine ordering, and notification semantics.

Failure accounting

  • Adds a sharding guard that lists all 26 folded suites and proves each is discoverable in broad packing with exactly one owner and no remaining focused step; existing tests prove retained focused suites stay excluded.
  • Moves the fish prerequisite for CmuxBundledBinPathIntegrationTests into the broad "Run unit tests" step, installing fish only on the batch that owns that selector.
  • Updates CLISSHSessionAttachAnchorTests documentation to reflect that CI now aggregates every test summary and rejects incomplete runs.

Summary by CodeRabbit

  • Tests

    • Updated automated test sharding to distribute regression coverage across timed app-host batches.
    • Added validation to ensure broad test suites remain discoverable and are not duplicated in focused checks.
    • Added prerequisite verification for Fish-based integration tests.
    • Expanded focused regression coverage for remote sessions, socket access, device encoding, feed handling, and related workflows.

  • Chores

    • Streamlined macOS CI by removing several dedicated regression steps and consolidating their coverage into broader test runs.