Cut poll-sweep replays at durable boundaries

VA
17h ago
+375-58

Problem

local::admission::poll_sweep (4 tests) fails on main in the gate run cargo test --locked -p acyclic-harness --all-features --no-fail-fast (baseline 66ddfd542). Disabling the coverage check does not fix them, so updating the pinned counts alone is not enough.

Breaking commit: #949 (d8d50b3aa). It adds spawned stream snapshot workers. Their task submissions add 2 to the Text and Compaction counts and 4 to EffectAndContinuation. They also make independent completion and blocking-pool boundaries interleave differently between replays. The sweep cut at the k-th boundary of any kind, so a replay could cancel at a different event than the baseline, and the prefix comparison failed.

Why the gate missed it: PR CI runs no harness Rust test target, so this lib test never ran before merge.

Fix

The sweep now cuts at operations, not boundary indices.

  • schedule::plan emits one cut for every occurrence of every durable boundary (model accepted/dispatch, native submitted, task submitted, tool accepted/dispatch). It also emits one for every occurrence of every operation boundary: native/task completion, and blocked/completed stream reads. A cut is (name, occurrence).
  • Every boundary name occurs the same number of times in every replay; the reviewed manifests pin those counts exactly. A planned cut therefore always fires at the same operation, however independent owners interleave. The plan is the same for every baseline: 168 + 408 + 378 = 954 cuts.
  • drive cancels at the planned (name, occurrence), drops the caller, drains the owners, and then the case recovers and resumes through the real paths. This now happens at completion cuts as well as durable cuts.
  • check_replay compares the replay's filtered durable iterator directly against the baseline's first ordinal durable boundaries (Iterator::eq, plus an exact-length check). It no longer collects vectors.
  • coverage::check_cuts requires the plan to be exactly every occurrence of every durable and operation boundary in the reviewed manifest, with no duplicates, and the sweep to run exactly that plan.
  • Coverage manifests are re-pinned to the counts measured after #949. admission_cut.json holds a durable ordinal (37).

Why the replay order varies between runs: the test setup keeps the physical worker handoff on purpose (native-runtime/src/testing.rs: "Keep the original queue admission, capacity wake, worker handoff and physical terminal"). Native jobs and shared blocking work run on real OS worker threads. When one finishes, its owner task wakes and races the other runnable owners to the boundary channel, so OS scheduling decides the arrival order of independent completions. The production code behaves the same way. This is not a race on durable state: the durable projection is identical across all 954 cuts in every run, which check_replay asserts at every cut. An intermediate version of this PR deduplicated "first operation of each kind per durable step" from the observed baseline. That plan inherited the arrival order (269 vs 270 cuts across runs), so it was replaced with the order-independent (name, occurrence) plan.

Tests

  • independent_reordering_keeps_every_operation_cut (proptest, 64 cases) randomly swaps adjacent boundaries that are not both durable. Every planned cut still fires at its operation with a replayable durable prefix, and durable cuts keep their exact ordinal.
  • plan_cuts_every_durable_and_operation_boundary: an exact plan for a hand-written trace, and the same plan for a reordered trace.
  • reordered_completion_negative_control: under reordering, the old all-boundary index cut diverges. Swapped or missing durable events are rejected.
  • retained_caller_at_a_completion_cut_is_rejected: a real-path negative control. It runs the Text scenario at a task:completed cut that the former durable-only plan never took. Unmutated, that cut recovers. With the mutated driver, which cancels but keeps owning and polling the caller, the run must panic with native-runtime's "new dispatch after cancellation". On bb1 it did, in all three runs.
  • broken_real_sweep_cannot_drop_a_cut (proptest): a sweep missing any planned cut, or a planner that drops one, is rejected.
  • bb1, three consecutive runs of cargo test --locked -p acyclic-harness --all-features --lib poll_sweep: 17 passed each time. The plan was identical every run (Text 168, EffectAndContinuation 408, Compaction 378). Suite times were 11.89 s, 8.64 s and 7.35 s, measured while the graphcoder tenant was at its 16-job compile cap on the same box. Clippy (--all-targets --all-features) reports nothing in poll_sweep.

Guarantee

  • Class: I/Is, order dependence. The sweep identified cut points by a global boundary index, so any reordering of independent work between the baseline and a replay moved the cut.
    • Structural change: a cut is named by the operation it cancels at, (boundary name, occurrence). That name is invariant under any interleaving, because per-name counts are fixed and manifest-pinned.
    • The plan covers every durable boundary and every operation boundary, so cancellation is exercised both before and after each submitted operation's result becomes observable.
    • check_replay is the only prefix comparison, and it runs at every cut.
  • Proof:
    • The sweep runs all 954 operation-aware cuts through drive, cancellation, drain, recovery and resume on the real paths. The plan is identical across runs.
    • Negative controls, all run:
      • Mutated ownership/cancellation: the caller is retained after a completion cut, and the run fails with "new dispatch after cancellation".
      • Dropped planned cut: rejected by check_cuts.
      • Reordered and missing durable events: rejected by check_replay.
    • Proptest over independent reorderings for every planned cut.
  • Entry points: every poll_sweep scenario (Text, Compaction, EffectAndContinuation, the indeterminate admission cut, the coverage manifests) goes through plan, drive, check_cut and check_replay.
  • Gaps:
    • blocking:submitted and blocking:completed are not cut points. They are the blocking-pool hops inside an operation; their owning native or task operation is cut at both submission and completion. Adding them gives 1,452 cuts, which ran 7.5 to 17.3 s on bb1, over the 10 s budget.
    • The suite takes 7.35 to 11.89 s on a loaded bb1, which is at the edge of the 10 s budget.
    • Both are open for the coordinator's decision.

🤖 Generated with Claude Code

Summary by cubic

Fixes local::admission::poll_sweep failing on main after #949 added spawned stream snapshot workers. Their task submissions changed the boundary counts and made independent completion and blocking-pool boundaries interleave differently between replays, so the sweep could cancel a replay at a different event than the baseline.

The sweep now cuts at the k-th durable boundary (submissions and external effects) instead of any boundary, and requires the replay's durable prefix to match the baseline's first k exactly; reordered or missing durable events are still rejected. Coverage manifests are re-pinned to the measured counts, and a proptest confirms every durable cut stays replayable under reorderings that break the old all-boundary cut.