feat(compare): score a trace replay against production's record of the same requests
Adds batchbench-compare-trace-replay, which scores a replay of Dynamo request traces against production's own record of the same requests.
It joins three inputs on each request's source row:
- the manifest's
metadata.requests(#42), which names each request's source row; - the agent's
--request-events-jsonl(#41), which holds the replay's client-side timings; - the trace rows themselves.
Over a scoring window of source time, it reports for both sides:
- requests and failures;
- first-token, time-per-output-token and end-to-end percentiles;
- the share that succeeded within a first-token bound and a time-per-token bound.
A failed replayed request counts against that share. --replay-gpus adds the replay's throughput per GPU. An optional production context file maps decode worker ids to worker sets and gives each set's GPUs. With it, production's figures are split by set, and with --model each set's throughput per GPU over the window is reported beside the replay's.
Only requests whose source arrived inside the window count. That lets an export that starts earlier warm the replay's caches before scoring begins.
Stacked on #42.
Checked end to end on a real manifest: a 1/16 session sample of GLM-5.3-Flash, 12 Nov 17:30ā19:00, scored over 18:00ā19:00.
- With an events file that replays production exactly, all 8,403 scored requests matched.
- Every replay figure equals production's, overall and per set.
- The per-set throughput agrees with an independent count: Isambard 0.1127 req/s/GPU, 110 output tok/s/GPU, 65.2% within 15 s / 33 ms.
Tests: 44 pass, 2 skipped (they need the compiled agent).
š¤ Generated with Claude Code