feat(telemetry): sort executed tests by cache tier and count miss reasons

Merged
OK
10d ago
+151-4

The run event had no per-test cache view, so "what % of tests run without the model" could not be read off PostHog, and a cache miss could not be told apart (first recording vs retry vs stale replay).

  • tests_code_only, tests_replayed, tests_assisted, tests_live: every executed test in exactly one tier, across its attempts. The four sum to tests_executed.
    • code-only: no agent step
    • replayed: every agent step self-finalized from the cache, no executor ran
    • assisted: the agent ran, cache carried at least part of a step
    • live: the agent ran every agent step, no cache help
  • agent_steps_uncached: agent.act steps with no cache record (cache off, executor cache: 'off'), so replayed + partial + missed + uncached = act steps.
  • agent_miss_reasons, agent_handoff_reasons: counts keyed by the runner's closed reason tokens (no-entry, retry, wrong-context, gap, target-not-found, ...), through plainToken.
  • model_calls_act|assert|extract|waitfor: where the model spend goes. Only act can replay, so assert-heavy suites never become deterministic.
  • testSteps (packages/e2e/src/telemetry/events.ts) now skips results with no attempt (unselected, never run, serial members whose attempts live on the group) and skipped serial members, matching summary.executed. Before, those would have landed in tests_code_only.

Dashboard e2e CLI Analytics already has tiles reading these: tests by cache tier, fully deterministic %, misses by reason, model calls by API.

Verified

Ran it locally: yes

  • Packed e2e + @e2e-dev/web into a scratch project with the web-benchmark agent suite and its committed cache, E2E_TELEMETRY_DEBUG=1 e2e run --config e2e.agent.config.ts (bad gateway key, so misses fail live):
Tests  7 failed | 27 passed (34)
Cache  26 replayed · 2 handed off · 4 missed

{"tests_executed":34,"tests_code_only":1,"tests_replayed":26,"tests_assisted":2,"tests_live":5,
 "agent_steps_replayed":26,"agent_steps_partial":2,"agent_steps_missed":4,"agent_steps_uncached":0,
 "agent_miss_reasons":{"gap":2,"no-entry":2},"agent_handoff_reasons":{"gap":2},"model_calls_assert":1}
  • Same project with two file positionals, so 26 tests unselected:
{"tests_discovered":34,"tests_selected":8,"tests_executed":8,"tests_code_only":1,"tests_replayed":7,"tests_assisted":0,"tests_live":0}
  • The new unit test (unselected result, serial member, skipped member) fails on the pre-fix testSteps with expected 10 to be 7.