feat(export): build manifests from Dynamo request traces
Adds a second input to batchbench-export-plans: --dynamo-traces <dir|file|glob> reads the request_end rows a Dynamo frontend writes with request tracing enabled (dynamo.request.trace.v1, gzipped JSONL) and writes the same schema v2 manifest as the prompt-chain source. With this, replays of Dynamo-served models don't need a separate gateway capture.
How it works
- Window. Trace files are named by upload time and can hold rows received much earlier, so every file is read and rows are windowed by
request_received_ms(--start/--end, plus the link window before--startfor link targets). - Retries. Rows that share an
x_request_idare attempts of one request. The exporter keeps the last attempt that finished and counts the others. - Sessions. It uses the same strict-prefix rule as prompt chains, applied to
input_sequence_hashes. Rows link only within one model, block size, principal and agent session id. Rows with differenttrace_block_sizeare never linked or merged. - Hashes. Dynamo hashes every full block, then the partial last block when tokens remain, so there are
ceil(input_length / trace_block_size)hashes. The exporter splits off the partial hash, so it is never used for linking or cutting; it only seeds the tail block. Rows that hash only full blocks also load. A row with more hashes than that counts asmalformed_hashes. - Blocks. Each prompt is cut wherever a chain of the same model and block size ends or diverges, using full blocks only. Each segment becomes a
userblock withseed= the segment's last hash andtokens=trace_block_size× its hash count, so shared prefixes replay with identical bytes. - Tail. The partial tail past the last full block becomes a final
userblock, not overhead. That keeps the replayed prompt at the recorded length, which matters when block sizes run to thousands of tokens. Sooverhead_tokensis 0. - Extensions. A request that extends the previous request in its trajectory repeats that request's emitted blocks, then a
liveassistant block of the previousoutput_tokens, then the new content. Only the newest reply islive. Earlier replies repeat as plain assistant blocks under their original seeds, and the agent replays an already-sent seed with the same bytes, real reply included. A reply that would not fit in the extension is generated (split where segments end) rather than replayed live, because a live block always replays the whole real reply.live_clampedcounts these; it is typical for reasoning models, whoseoutput_tokensinclude reasoning that clients don't send back. - Branches. Any other request in a trajectory starts with the blocks already sent for the longest prefix it shares with an earlier request.
reset_beforeclears the agent's cache, so generated blocks come out the same from their seeds, but copies of earlier replies are generated rather than real. - Optional join.
--analytics-table(needs--clickhouse-url) or--analytics-rows-jsonljoins onx_request_id(--analytics-id-column, defaultx_request_id). The analytics side is bounded by time, and the id is compared as a string. A match supplies the principal (user_id, elseapi_key_id) and the alias (in trajectorymetadata). Without a join, the exporter runs on the traces alone, and--principal-idis rejected. - Stats. The summary line adds trace counters: files, unreadable files, skipped rows by reason, duplicate attempts, join matches, live and clamped blocks, and rows per block size.
docs/export-plans.md documents the source, the join, and the fidelity lost compared with prompt chains: no roles, no tools array, template overhead merged and double counted, block granularity, large blocks hiding divergence and splitting sessions, approximate reply length, no stream/max_tokens, and timing that starts at Dynamo.
link_parents now groups by a link_key property (the principal, for chain records). It also finds candidate parents with a set intersection, because trace chains run to thousands of hashes. The behaviour is unchanged for chain records.
Testing
python -m unittest discover -s tests: 37 tests pass, including the existing ones. The new tests generate cumulative hashes in the test, in the production ceil shape. They cover shared prefixes, divergent branches, partial tails (ceil, floor and exact shapes, plus a turn that fills the previous partial block), three-turn sessions, branch reuse, a 1056-token block size, two block sizes with colliding hashes, a row received before its file's upload time, a truncated file, retried attempts, and rows with and withoutx_request_id, joined and unjoined.- The exported manifest dry-runs in
batchbench-agentwith both live blocks replayed. - A synthetic scale check (19.5k rows, 12M hashes, 98 files) ran in about 3 s with about 200 MB RSS.