feat(report): record each step's cache outcome (written, kept, evicted and why)

Closed
AK
14d ago
+286-37

Closes #939.

What and why

  • StepCacheInfo says how a step read the cache (mode, reason) but not what the attempt then did to the recording. A self-healing CI loop needs that to know which tests to run a second pass for, and today it has to diff .e2e/cache/ against git.
  • Adds two optional fields, as proposed in the issue:
    • outcome: 'written' | 'kept' | 'evicted'
    • outcomeReason: kept is confirmed or unchanged; evicted is repaired, failed-after-replay, not-replaced or unconfirmed; absent on written.
  • Additive: both fields are optional and only appear in read-write mode when the attempt changed or confirmed an entry. A read-only cache, an interrupted attempt, a step with no verdict, or nothing to record and nothing stored leaves outcome absent. No schema version bump.
  • The report only claims what the store actually did: a custom store without delete, a rejected delete, or a failed write reports no outcome.

Each outcome comes from the place that already makes the decision:

flowchart LR
  S[step settles] --> C["StepTraceSession.conclude()"]
  C -- "end-mismatch repaired" --> E1["evicted: repaired"]
  C -- "failed after replay" --> E2["evicted: failed-after-replay"]
  C -- "passed, nothing to stage, entry read" --> E3["evicted: not-replaced"]
  C -- "staged write / keep" --> A[attempt ends]
  A --> F["flushStagedTraces()"]
  F -- "not verified, implicated" --> E4["evicted: unconfirmed"]
  F -- "keep, verified" --> K1["kept: confirmed"]
  F -- "write, same flow stored" --> K2["kept: unchanged"]
  F -- "write, verified" --> W[written]
  • Step-time evictions: packages/e2e/src/agent/step-cache.ts, evict(reason) sets the outcome on the step's cache info once the delete has settled. conclude() runs before finish() attaches the details, so these reach both the report and agent.act()'s result.
  • Attempt-end settlement: flushStagedTraces (packages/e2e/src/cache/context.ts) now returns { stepIndex, outcome, outcomeReason? }[], and packages/e2e/src/run/execute.ts merges them into record.steps[stepIndex].cache right after the flush. agent.act() returns before the attempt settles, so its result carries only the step-time evictions. The docs say so.
  • Wire change: report-v1.schema.json gets both enums plus allOf rules (reason only with an outcome, no reason on written, each reason only with its own outcome). Both fixtures and a schema test pin those rules.
  • Docs: a new "What a run did to a recording" table in docs/cache.mdx, plus docs/reference/agent.mdx and skills/e2e/references/agent.md.
  • One existing unit test, evicts an entry that did not serve a pass with nothing to record in its place, recorded the replayed tap after begin(), so it was going down the repair branch instead. Its host now records the tap through the grammar, as the real dispatch does, and the test asserts not-replaced.

Verified

Ran it locally: yes

  • A real e2e run against the browser fixture app with a scripted two-tap executor and cache: 'read-write'. Run 1 records, run 2 replays, and run 3 tightens the assertion so it fails. This is step.cache from each run's report.json:
run 1, record:          {"mode":"missed","reason":"no-entry","replayedActions":0,"totalActions":0,"outcome":"written"}
run 2, replay:          {"mode":"self-finalized","replayedActions":2,"totalActions":2,"outcome":"kept","outcomeReason":"confirmed"}
run 3, assertion fails: {"mode":"self-finalized","replayedActions":2,"totalActions":2,"outcome":"evicted","outcomeReason":"unconfirmed"}
  • The integration suites trace-cache-replay and agent-trace-cache now assert the outcome end to end: kept/confirmed on whole replays, written on re-recordings, evicted/unconfirmed when a later assertion fails, and no outcome when the failure is a model that never answered or the cache is read-only in CI.
  • Done on Windows. The full e2e package run has failures in 26 files on Windows (CRLF line endings, \ paths, POSIX file modes, git/telemetry environment). All 26 fail the same way on main, and none of them touch the cache or the report. pnpm docs:check was not run locally, because it needs the Mintlify docs workspace installed.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: not one. Both fields are optional additions.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures. (sdk-types.ts has no StepCacheInfo assertions, so nothing changed there.)
  • Engine contract change (e2e/engine): not touched.