feat(telemetry): sort executed tests by cache tier and count miss reasons
Merged
OK
10d ago+151+151+012345678901234567890123456789-4-4-0123456789
The run event had no per-test cache view, so "what % of tests run without the model" could not be read off PostHog, and a cache miss could not be told apart (first recording vs retry vs stale replay).
tests_code_only,tests_replayed,tests_assisted,tests_live: every executed test in exactly one tier, across its attempts. The four sum totests_executed.- code-only: no agent step
- replayed: every agent step self-finalized from the cache, no executor ran
- assisted: the agent ran, cache carried at least part of a step
- live: the agent ran every agent step, no cache help
agent_steps_uncached:agent.actsteps with no cache record (cache off, executorcache: 'off'), so replayed + partial + missed + uncached = act steps.agent_miss_reasons,agent_handoff_reasons: counts keyed by the runner's closed reason tokens (no-entry,retry,wrong-context,gap,target-not-found, ...), throughplainToken.model_calls_act|assert|extract|waitfor: where the model spend goes. Only act can replay, so assert-heavy suites never become deterministic.testSteps(packages/e2e/src/telemetry/events.ts) now skips results with no attempt (unselected, never run, serial members whose attempts live on the group) and skipped serial members, matchingsummary.executed. Before, those would have landed intests_code_only.
Dashboard e2e CLI Analytics already has tiles reading these: tests by cache tier, fully deterministic %, misses by reason, model calls by API.
Verified
Ran it locally: yes
- Packed
e2e+@e2e-dev/webinto a scratch project with the web-benchmark agent suite and its committed cache,E2E_TELEMETRY_DEBUG=1 e2e run --config e2e.agent.config.ts(bad gateway key, so misses fail live):
Tests 7 failed | 27 passed (34)
Cache 26 replayed · 2 handed off · 4 missed
{"tests_executed":34,"tests_code_only":1,"tests_replayed":26,"tests_assisted":2,"tests_live":5,
"agent_steps_replayed":26,"agent_steps_partial":2,"agent_steps_missed":4,"agent_steps_uncached":0,
"agent_miss_reasons":{"gap":2,"no-entry":2},"agent_handoff_reasons":{"gap":2},"model_calls_assert":1}
- Same project with two file positionals, so 26 tests unselected:
{"tests_discovered":34,"tests_selected":8,"tests_executed":8,"tests_code_only":1,"tests_replayed":7,"tests_assisted":0,"tests_live":0}
- The new unit test (unselected result, serial member, skipped member) fails on the pre-fix
testStepswithexpected 10 to be 7.