Stop rescanning durable object history on every put

Merged
VA
13h ago
+1058-98

Problem

Default-cap goal continuations spend growing CPU on historical durable-object size validation. On the shipping current-thread HTTP/headless path, one interval grew from 131,699 body checks near turn 3 to 3,284,090 near turn 33. Every deferred put rescanned the complete validated object/part prefix.

Design

Validate only the pending body in both ordinary and native durable batches. The validation function no longer accepts State. Centralize the immutable per-body limit in prepare_inline before the already-persisted fast path, covering inline multipart parts and composite bodies as well. Recovery already validates the retained prefix against the immutable header limits. Multipart preparation, relocation and delta encoding now consume persistent-map differences rather than visiting retained parts. Extend the existing byte total to include staged parts and update it from those same differences, preserving aggregate quotas, publication, rollback and recovery semantics.

Deletions

Delete the full-state validation iterator, retained multipart preparation/encoding/relocation walks and full staged-part quota sum. Reuse the existing byte-total scalar for objects plus staged parts; no new cache, synchronization primitive or store. Net line delta: +1,058/-98, net +960 across 18 files; production changes are +30 net lines, with the remainder in verification.

Decisions

The root is durable put-batch validation, rather than a goal-specific shortcut. Both native and ordinary callers share the pending-body check. No closure-proof, request-history or segment-decoder changes overlap other lanes. Temporary profiling is outside the committed tree. Test-only hooks restore each removed multipart scan inside its real commit phase; the complete batch measurement uses a scoped fixed clock because protobuf timestamp widths otherwise change allocation bytes. Neither counters, restored scans nor the clock override exist in release builds. The two default-limit goal scenarios and their guards remain unchanged in #600/#700.

Guarantee

  • Class: H (repeated history work), with I-Ip/I-Is regression oracles. The validation API takes one immutable body and cannot walk State. Multipart mutation stages are driven by persistent-map differences, so unchanged subtrees do not enter validation, encoding, relocation or byte accounting. One persistence choke point still validates all newly stored object/part bodies, including newly composed persisted bodies; recovery establishes the retained prefix.
  • Proof: exact four production StoredBody::len calls per fresh durable put at 16 and 32 puts through both public batch APIs; restoring the removed scan violates that same constant-work predicate. Shrinking independently initialized old/new validation comparison checks full bytes, exact quota errors, ordered empty observer traces, exact physical allocation fields and one candidate body read. The old scan allocates its tree cursor above 16 keys; the candidate allocates zero. Shrinking mixed accepted/rejected native/ordinary batches check all reopened bytes and rejected-key absence. Multipart inline/external boundaries check failed overwrite rollback, journal/sequence preservation, complete/reopen and already-local composite rejection. New N=16/2N=32 retained multipart-prefix oracles run both public batch APIs: exactly four body reads per unrelated write, two to append one part, and zero reads/allocations in unchanged inline preparation. Restoring the exact old retained-part preparation loop adds N reads and fails the same constant-work predicate. A shrinking independent byte-total model covers part insertion, replacement, deletion, upload removal and quota rejection; real public operations cover overwrite/abort/completion plus cold reopen. Omitting removed-part accounting is the total-oracle negative control. The complete public batch call is also measured at N=16/2N=32 retained parts through both APIs: exactly one encoding and one relocation visit, with exact equality of all six allocation fields at both sizes. Controls restore the eager bodies() encoding loop inside difference() and the full mutable relocation loop inside place(), separately. Each adds exactly N visits and independently exceeds both allocated-count and allocated-byte bounds; unchanged expectations still verify committed writes and cold-reopened multipart state.
  • Entry points: MemoryObjects::put_batch → put_durable_batch_locked; ObjectsProvider::put_batch_native → put_locked_native; ordinary/native mutate closures; object writes, copies and multipart parts/completion through Journal::prepare_inline; recovery apply_objects/apply_upload_parts. No parser, concurrency protocol or lock order changes.
  • Gaps: none. The coordinator explicitly deferred the macOS allocation-oracle rerun listed under Live steps; Linux proves the class and Windows GNU compiles all touched code.

Tests run

At signed 54f72754c5223049527b4ce4eac4dd2102409c48, bb2 job1792322144180-1346351 passed all nine body-validation proofs in0.975s; multipart recovery0.056s, batch quota0.019s, atomic rollback0.015s and journal replay0.191s. Objects Windows GNU local lib/tests passed13.896s. Native build13.544s and queue2s are excluded. The new complete-batch oracle includes admission, mutation, encoding, relocation and durable commit, with fixed input timestamp widths. Every unrelated write makes1 encoding visit and1 relocation visit at both N16 and2N32. Ordinary batches allocate43 times/12,082 bytes; native batches38 times/11,711 bytes, flat at both prefix sizes. All six allocation fields are compared exactly for all four writes at both sizes.

Restoring eager encoding adds16/32 encoding visits and changes ordinary allocations to75/108 (13,746/15,426 bytes), native70/103 (13,375/15,055 bytes). Restoring the old relocation walk adds16/32 relocation visits and changes ordinary allocations to45/49 (14,834/18,754 bytes), native40/44 (14,463/18,383 bytes). Both controls independently fail the visit and heap predicates through both APIs. The earlier four-length-read, staged-byte accounting, rollback, composite-size and recovery proofs remain green. An initial compile attempt used an unstable TLS convenience method (corrected); a first runtime attempt correctly caught variable-width real timestamps in the exact byte comparison (corrected with scoped fixture time). Neither failed attempt is reported as passing.

Exact gate: Q:/lanes/bs.ps1 python3 -u -c (Get-Content -Raw Q:/lanes/gc-e2e-1015-full-batch-gate.py). It builds cargo test --locked -p acyclic-objects --lib --features local --no-run --message-format=json once, runs the retained binary with body_validation, multipart_recovery_preserves_parts_composite_ranges_and_completion_retries, one_durable_batch_preserves_independent_validation_and_quota_failures, staged_histories_keep_atomic_rollback_and_exact_quotas, and journal_deltas_replay_and_recover_prefix_states filters, then runs cargo check --locked -p acyclic-objects --lib --tests --features local --target x86_64-pc-windows-gnu.

Earlier paired qualification: at signed ca9b6e6dd31f85a4137e28c551c5a141361522d3, bb2 job1792317351067-1023964 passed: four new proofs in0.740s, independent batch quota regression in0.012s, replay property in0.157s. Real HTTP/headless before4.590s → after2.578s (33 requests); durable put-batch spans990ms →269ms. Windows GNU Objects lib/tests with local feature passed22.751s. Native builds5.424s (Objects) and55.221s (diagnostic e2e), excluded from runtimes. The committed source excludes all temporary profiling modules; paired diagnostics and logs are retained outside the product tree in goal-validation-hpo71zy9. macOS real local/native batches: exact four reads at 16/32, reverted scan 4..35; quota/reopened-byte property and multipart test pass. The first differential run rejected my incorrect zero-allocation expectation for the old tree iterator, now corrected from its implementation; it is not reported as a passing property.

Paired Linux command (builds once, reuses the retained baseline binary): Q:/lanes/bs.ps1 python3 -u -c (Get-Content -Raw Q:/lanes/gc-e2e-goal-validation.py). Baseline/current use independently initialized workspaces, real loopback Chat HTTP, shipping connect_local/run_headless entrypoints and current-thread runtime, 33 model requests. This is the headless tier that exposes the growing history cost; isolated smaller goal cases did not reach the default-cap workload. Full retained goal-family gates remain acceptance requirements on #600/#700.

Live steps

User/coordinator-deferred macOS allocation-oracle rerun: ivar now requires a Tailscale authentication check while the user is asleep. The coordinator will run the corrected oracle once access returns. Exact N/2N ordinary/native work, multipart rollback and reopened-byte properties already passed on macOS at production-identical 12bbca58e; the first allocation oracle correctly detected the old iterator's one-frame allocation, and its expectation is corrected at ca9b6e6dd. No macOS allocation-oracle PASS is claimed.

From a checkout of 54f72754c on ivar: CARGO_TARGET_DIR=$HOME/mac-target/gc-e2e RUSTC_WRAPPER=/opt/homebrew/bin/sccache cargo test --locked -p acyclic-objects --lib --features local body_validation -- --nocapture. Reuse that single lane target and delete it after the run, per the current ivar disk rule.