#915 Publish each response delta as its live chunk in one fenced commit

Depends on #839. This branch sits on #839's current head (a3abd7151), so the diff includes #839 until that PR merges. The commit this PR adds is 9c29db458. It rebases #915 onto #839's structure (the retained VerifiedSummary and the task-wide chunk sequence) so neither fix is made twice.

Summary

This fixes F6 for streamed text and reasoning, where each delta cost about 32 KB of state. The old per-delta path staged a private JSON file (object writes, publication and per-object fsync), then appended an observation that named the file.

A task-owned execution journal now records each Response text or reasoning delta as its live chunk. The chunk record (.tasks/{task}/turns/{turn}/chunks/{execution}/{sequence}, raw delta bytes, channel Text or Reasoning) commits in the same Head-fenced, multi-record transaction as the observation that names it:

• No private file: nothing is staged, so there are no object writes and no fsync per delta.
• No growth: growing state is never rewritten and no transcript snapshot is taken.
• The observation: a FileRef on the task files volume at the chunk path, version task-chunk. Its descriptor covers the canonical JSON view {"delta":..,"kind":"content"|"reasoning"}, which loads rebuild from the chunk bytes and then verify.
• Admission: the JSON just encoded (pending) is verified instead of re-read.

Review findings addressed (#915 CHANGES)

• P1: native publication was quadratic. Each append replayed the whole journal (observations.rs:331).
  • Now every append advances #839's retained VerifiedSummary by one record.
  • Native linearity is a required assertion: native_delta_publication_on_local_disk_is_linear_with_exact_counts runs on a real LocalStream and Fs::local in a tempdir, with exact counts at N=8 and 2N=16.
  • Its negative control runs the same deltas while every append replays the whole journal (with_append_scan). That control fails the same linear bound.
• P2: reopen reordered chunks across executions.
  • A delta's chunk now takes the task-wide publication sequence from #839, so it is not reordered by execution UUID.
  • The observation must name its chunk before the append, so the journal names the next sequence (next_chunk_key).
  • prepare_named_chunk checks the sequence again under the index lock. A peer publication in between returns SequenceConflict, and append_delta renames the delta and retries.
  • Retry identity covers (step, purpose, volume, descriptor), not the assigned sequence, so an exact retry stays idempotent.
  • Coverage for live publication versus reopen, with reverse-sorted IDs: delta_chunks_follow_publication_order_across_reverse_sorted_executions uses peer execution ff… and then OP = 02…, and the reopened snapshot is [(0, ff, peer), (1, 02, working)]. The test also reads the observation's chunk back after reopen.
  • a_peer_publication_after_naming_the_next_chunk_is_a_sequence_conflict has a negative control: the unchecked prepare_journal_file accepts the key that would repeat sequence 0.
  • #839's chronological reconstruction proptest now publishes through next_chunk_key plus prepare_named_chunk. prepare_next_chunk is deleted.

Class and why it can't recur

Class: per-delta work, storage or allocation that grows with the transcript or far exceeds the delta, plus publication order that depends on identifiers instead of publication position.

Why it can't recur:

• Chunk-only deltas: a task-owned journal rejects any Response delta observation whose ref is not its chunk ("task response delta must be recorded as its live chunk").
• One chunk per sequence: a chunk key commits at most once (ImmutableConflict). Sequences are task-wide and must be the next one at commit time (SequenceConflict), so neither reopen nor live readers can reorder or repeat them.
• Atomic publication: the chunk and its observation are one fenced append, so the cursor sees each delta exactly once and in order.
• Retained summary: each append advances it by one record. The on-disk linear test fails if any per-append replay comes back.

Proofs (debug on bb1 and Windows; release per OS below)

• chunk_tests::publication::delta_publication_is_exactly_linear_in_bytes_work_and_allocations covers the whole publication path, non-native and native ledger.
  • Exact pins at N and 2N for:
    • stream writes, records and bytes;
    • ledger records and bytes;
    • object writes and bytes;
    • index items;
    • allocation count and bytes;
    • native allocation, copy, encode and hash counters.
  • Every durable byte doubles. Stream bytes are a constant 995 per delta.
  • It checks against an independent run of the former private-file path: identical read-back JSON and identical errors.
• native_delta_publication_on_local_disk_is_linear_with_exact_counts (new, real native backend):
  • Exact counts at N and 2N.
  • Items, SDK allocations, and ledger records and bytes double exactly.
  • Native counters stay within a constant of twice N. The byte slack is one geometric reallocation of the stream provider's retained-commit index, traced to MemoryStream::reserve_retained at 16 B per entry. The counter slack covers varint widths.
  • Negative control: whole-journal replay breaks the bound.
• whole_transcript_rewrite_negative_breaks_the_linear_pins: the same path plus a whole-transcript rewrite per delta fails the pins, with identical content.
• chunk_projection_has_exact_linear_work_and_allocations: each chunk costs exactly 6 items: one record read, one sequence lookup and four index operations.
  • Non-native: 23 allocations and 2,812 B per chunk.
  • Native: 24 allocations and 3,014 B per chunk.
  • The copy negative control is kept.
• entire_append_has_exact_linear_work_with_prefix_replay_negative (#839's test) is re-pinned. Its prefix-replay negative still fails.
• chunk_transaction_survives_each_crash_cut and negative_control_split_commit_loses_the_chunk go through production append_delta.
• Measurement fixture: peer chunks advance the task sequence to 10 first, so every measured chunk path at N and 2N has a two-digit sequence. Without this, decimal width alone would break exact doubling.
• graphcoder: all tests pass, including live_progress (2/2).
• Focused suites: 24/24 pass in debug on Linux (bb1) and Windows. The release chunk suite passes 13/13 on both.
• Harness lib (bb1): 1,105 passed and 25 failed. Twenty of the failures also failed on the base runs (memory summary_tests, future_sizes, pinned_read, journal_accounting and others). The other five are the same core state namespace is in use by another store ownership race: the tests share one harness id under parallel runs. None of them run the journal paths this PR changes.
• Windows compile proof: cargo check --target x86_64-pc-windows-gnu with the grpc feature passes (rc 0).

Per-delta cost, N=8 (release; former private path → live chunk)

	stream bytes	object writes	allocations	allocated bytes
non-native, former	7,128 (8 rec)	—	21,472	7,831,301
non-native, chunk	7,960 (16 rec)	0	2,439	605,799
native, former	7,128	154 (77,081 B) + 72 ledger records	23,310	9,148,405
native, chunk	7,960	8 fenced appends, 0 object bytes	3,204	1,926,145

Allocated bytes fall 12.9× on the non-native path and 4.75× on the native path. At 2N=16 the native chunk path costs 6,170 allocations and 3,805,041 B. Before this rebase it cost 30,294 allocations and 7,246,191 B, which was quadratic.

Native on real disk (LocalStream + Fs::local, release)

	items	allocations	allocated B	native alloc / copy / encode / hash	ledger
N=8	120	2,934	1,993,558	2,225 / 64,472 / 78,392 / 87,608	16 rec / 7,960 B
2N=16	240	5,868	4,009,214	4,452 / 141,520 / 156,912 / 175,248	32 rec / 15,920 B
replay negative N=8	176	10,454	3,404,236	7,549 / 174,108 / 208,304 / 179,688	16 / 7,960
replay negative 2N=16	480	30,911	8,140,936	21,513 / 447,768 / 585,952 / 452,336	32 / 15,920

Measurement command (paired: the former path, the chunk path and both negative controls run in the same binary)

cargo test --locked --release -p acyclic-harness --lib --features filesystem-local,native-execution,test-support -- execution_journal::chunk_tests journal_wrapper_removes_exactly_one_allocation --nocapture --test-threads 1

Per-OS release numbers

OS	result	non-native chunk N=8	native chunk N=8 / 2N	disk native N=8 / 2N
Linux (bb1)	13/13 pass	2,439 alloc / 605,799 B	3,204 / 6,170 alloc	2,934 / 5,868 alloc
Windows (local)	13/13 pass	2,439 / 605,799 B	3,204 / 6,170 alloc	2,934 / 5,868 alloc
macOS (ivar)	pending: ivar needs Tailscale re-authentication	—	—	—

Linux and Windows produce identical numbers. The only debug-versus-release difference is that debug builds of the local stream make one more 104 B allocation per delta, and the disk pin selects on debug_assertions. The macOS arms keep the previously measured Darwin offsets (non-native +7 allocations / +600 B; native chunk +1 allocation / +64 B per record). They will be re-measured when ivar is back.

Findings prevented

• F6: about 32 KB of state per streamed text chunk; the store grew from 183 MB to 266 MB in one live pass.
• P1: quadratic native publication per append.
• P2: chunks from different executions reordered on reopen, which a live reader would have seen in a different order.
• Crash safety: a delta observation can no longer name a file that commits separately, so a crash cut cannot leave a missing or orphaned chunk.

Deletions

214 lines deleted (1,354 added) against #839's head. This includes the per-chunk split constant pins, the projection-copy path for task deltas, and prepare_next_chunk together with its Sequence::Next mode.

Decisions

• Name the sequence first, then verify it under the lock. The alternative was to let the volume pick the sequence at commit. But the observation is digested and carries the chunk's path, so it must know the sequence before the append. A conflict retry is cheap because it is O(1) and happens only under a concurrent peer.
• Retry identity excludes the sequence. If an append commits but its acknowledgement is lost, the retry may name a later sequence. The digest of (step, purpose, volume, descriptor) keeps that retry idempotent, so the delta is not recorded twice.
• Raw bytes plus channel in the chunk. ClientViews and the TUI consume the bytes directly. The JSON view is rebuilt and verified against the descriptor.
• ExecutionJournal::record_model is an optional default method. Journals that are not task-owned keep stage_json plus append. Approvals code is untouched.
• Follow-ups:
  • staged_objects/publication.rs:180 (per-object fsync rather than put_many) is the next stream-c PR.
  • Deltas over 64 KB need a multi-record chunk append.
  • A Loom or proptest model of atomic publication, plus gc-e2e stream scenarios.
  • walk.rs belongs to perf-h.

🤖 Generated with Claude Code