PR #860 - acyclic-labs/sdk

Class

H: unbounded or repeated work, copies and retention (error-classes rank 3). This PR covers the bounded follow behind Query::TaskView and Query::TaskOverviews (task_volume/recent.rs).

Invariant

The recent display window is a private, metadata-only structure in task_volume/recent/window.rs. It holds key, order, kind and size, with a per-kind (order, key) index and a running byte total. Only one put / remove / evict path changes it, and that path folds the snapshot and every change. The cost of a change depends on the files in that change and the evictions it causes, not on how many files the window retains. The window never holds a payload.

What the old path did on every change:

For each of the eight display kinds, it rescanned the whole retained window to count that kind, and scanned it again to find the oldest entry.
It re-summed retained bytes before every global eviction.
It rebuilt every retained key into a BTreeSet to work out the removals.
It kept a FileValue clone of every retained file, so payload storage stayed alive after the consumer dropped it.

Why it cannot recur

The window's fields are private to recent/window.rs. observe can only call Window::new and Window::apply.
Per-kind counts come from index length, and byte totals are kept by put/remove. No code path can scan or sum the window.
Removals are collected while entries are removed. An entry counts as visible before the change only if its generation predates the change, so no before-set is needed and the removed list reuses the change's own removed allocation.
The deleted code is gone: the scanning trim, the BTreeMap<FileKey, (u64, FileValue)> window and the per-change key set.

Semantics preserved exactly

These are checked against an independent copy of the old implementation:

Eviction order: per kind 0..7 first, then the global oldest (order, key). Equal orders evict the lower key first.
A key is reported removed only if it was visible before the atomic change and is absent after it. Reported keys are in key order.
Edge cases covered:
- replacement, and oversized replacement (removes the file);
- explicit removal of a key, then reinsertion in the same change;
- removal of a key that isn't present;
- a new entry evicted in the same batch;
- unclassified kinds, which have no per-kind bound;
- reasoning chunks, which are classified by channel;
- duplicate keys in a snapshot (later wins; oversized duplicates are ignored);
- errors in the change stream.
Snapshot files keep their typed identity order. They are now selected in place (retain + sort) instead of cloned.

Verification

All tests are in task_volume/recent/window/tests.rs and run through the real recent::observe / Window production path.

Independent baseline and full equality. window_matches_independent_old_window is a proptest. It covers dense, sparse, tied, unknown and oversized cases, duplicates and stream errors. differential::pair initializes each side separately, with fresh payload copies per side. The test compares snapshot bytes, every change's file/files/removed/cursor, exact error strings, and the ordered insert/remove trace recorded on the production path.
Hand-checked case. window_reports_only_keys_visible_before_each_change asserts exact expected output for the tricky orderings above, and also checks equality with the old window.
Exact work and allocations. window_change_work_is_independent_of_retained_files runs three changes on windows of N=64 and 2N=128 files:
- an unrelated checkpoint: exactly 0 allocations;
- an explicit removal: exactly 0 allocations;
- an oversized replacement: exactly 1 allocation of 4 * size_of::<FileKey>() bytes.
The ordered trace is exactly [remove 0, remove 1] at both sizes. For comparison, the independent old path's checkpoint allocations are positive and grow from N to 2N.
Negative controls.
- rebuilding puts the old per-change rebuild of all retained keys back on top of Window::apply. It fails the same exact assertion.
- In window_retains_no_payload_custody, holding the old window's payload clones beside the metadata makes all 16 payloads shared, against 0 for the new window. The old baseline leaves 15 snapshot payloads and the replacement shared.
Existing coverage still passes. Filters task_volume:: (16 tests), client::task_files (4) and client::commands (6) all pass.

Review fixes (rev-860-verdict)

A replacement that survives keeps the replaced entry's pre-change visibility instead of being queued as a removal. Only actual removals are pushed, into the change's own removed buffer, so an ordinary rewrite (such as a status update) allocates nothing. window_change_work_is_independent_of_retained_files now pins a fourth change, a surviving replacement, at exactly 0 allocations, with the exact trace [remove 0, remove 1, remove 2, insert 10] at both N and 2N. The negative control queuing puts back queuing the replaced key, and it fails that pin.
The baseline builds the old window with the original drain/filter/map/collect, verbatim. It records the trace that construction implies separately, outside the measured construction. The per-OS numbers below were re-measured at head 3f0e80923 against this verbatim baseline.

Gates at the new head:

bb1: task_volume:: 16/16, client::task_files 4/4, client::commands 6/6. Build plus run took 694 s.
Windows GNU proof: 2 m 28 s.
Window tests: 5/5 on macOS (0.03 s) and on Windows (0.02 s).

Paired measurement

Command (debug build, same source and instrumentation; old and new run in the same test):

cargo test --locked -p acyclic-harness --lib --features filesystem-local,native-execution,test-support,grpc task_volume::recent -- --nocapture

Workload: 512 retained files, then 128 one-file changes, each evicting the oldest file. Outputs are asserted equal.

OS	old allocations / bytes / time	new allocations / bytes / time
Linux (bb1)	7,246 / 13,512,160 / 16.4 ms	338 / 232,856 / 0.69 ms
macOS arm64 (ivar)	7,246 / 13,512,160 / 28.9 ms	338 / 232,856 / 0.71 ms
Windows MSVC (local, lb.ps1)	7,246 / 13,512,160 / 15.1 ms	338 / 232,856 / 0.51 ms

Timings are printed and never asserted. Allocation counts are identical on all three OSes. All 5 window tests pass on each: macOS 0.03 s (build 8 m 34 s), Windows 0.02 s (build 14 m 54 s). The change is platform-independent: no cfg-specific paths, and no new caps or limits.

Timings

bb1 lib test build plus the task_volume:: run: 578 s, almost all of it build. The tests took 0.03 s.
Windows GNU proof (cargo check --locked -p acyclic-harness --lib --tests --features filesystem-local,native-execution,test-support,grpc --target x86_64-pc-windows-gnu) passed: 5 m 26 s.

Historical findings prevented

rev-626-verdict3.md:3: a live projection rescanned retained task files on every chunk.
rev-626-verdict.md:8: repeated scans over retained entries.
rev-626-verdict-d.md:1: follow retention must stay bounded throughout. The bound is unchanged; it no longer costs O(window) per change or holds payloads.

Deletions

68 production lines were removed from recent.rs: the scanning trim, the payload-holding Window alias, and the per-change before-set and clones. The new production window is 152 lines.

Decisions

Per-kind ordering is one BTreeSet<(u64, FileKey)> per slot (8 kinds plus unclassified). The global oldest is the minimum over 9 first() probes. FileKey clones are Arc reference counts, so they don't allocate.
file_kind (unchanged classification) is shared with the oracle. Everything this PR changes in the window and its fold is re-implemented independently in the oracle.

🤖 Generated with Claude Code