fix(cache): report the models behind replayed recordings

Merged
AD
9d ago
+401-28
  • Save one optional recordedBy: { provider, model } from the resolved configured model, redacted once by the recorder. Cached steps expose it in step.cache.recordedBy and the trace page.
  • Show replayed · recorded with provider/model only for a known mismatch, once per test and recording inside its file block. Compare only the agent named in the header, including when a step pins another agent.
  • Preserve replay keys, per-member saves, the existing header and AI summary. Ignore malformed, blank or oversized attribution without losing a cache hit. Older recordings stay compatible.
flowchart LR
  A[Record with A] --> B[Replay with B]
  B --> C[Report A and show the mismatch]

Addresses #1000. Maintainer confirmation of this approach remains pending, including whether a repaired step should report the consulted recording's A while its replacement entry names configured model B.

Verified

Ran it locally: yes

Built CLI on a real Chromium counter app: node ../../packages/e2e/dist/cli/bin.js run --config e2e.model-probe.config.ts from apps/testbed, using controlled AI SDK responses.

mainthis branch
Replay under B has no record of configured model AJSON and trace page identify A; console names A inside the file block
model fake-loop/model-B
✓ |web| model-probe.e2e.ts (1 test)
  ✓ model provenance
    agent.act "increment the counter twice" replayed · recorded with fake-loop/model-A
  • Recording A made two model calls. Same-model, different-model, legacy, malformed and blank-metadata replays made zero calls and preserved recording bytes. An earlier serial member's recording survived a later worker crash.
  • The actual decision executor recorded configured text model A, replayed without a notice under A, and showed one under B. With no configured text model, recording and replay omitted attribution. Replays made zero decision calls.
  • Regression tests cover mixed-model executors, unreported model identities, retry and serial deduplication, per-call pinned agents, and the 300-character boundary. Oversized attribution still replays with zero model calls.
  • --no-cache --ai-trace matched main's 4,438-character prompt, 23 tools and action sequence. Testbed: 78 passed. Browser benchmarks: 91 deterministic tests and 26 replay tests, zero replay calls and all 30 committed entries unchanged. The stress suite produced its expected failures.
  • The repeated-label CLI case replayed two distinct recordings across two attempts with zero executor calls, two notices and unchanged recording bytes.
  • Previewed all three updated docs pages. The cache guide only reorders the reuse rule before the evaluation advice, with no prose added or removed.

At 0710b120, pnpm check and all 4,556 tests passed. A fresh-context review found no critical or important issues.

Six randomized paired control-inventory benchmarks against main e6260535 matched all outcomes and counters, including zero model calls:

Scenariomain medianbranch medianresult
Seven control-inventory tests7.43 s7.39 sSame within 5%

External providers were not called; four derived-input scenarios and three model judgments need a live provider. Screenshots and video are retained locally because GitHub attachment uploads require upstream write access.