feat(telemetry): count agent steps by api and tests that used the agent

Merged
OK
13d ago
+88-0
  • steps_agent counted every agent step together and the run event had no per-test view, so PostHog could not say what share of tests is deterministic end to end.
  • Adds steps_agent_act, steps_agent_assert, steps_agent_extract, steps_agent_waitfor by the step's api, and tests_with_agent_steps plus tests_with_act counted once per result and once per serial member by index across attempts (packages/e2e/src/telemetry/events.ts).
  • Share of deterministic tests over a period: 1 - sum(tests_with_agent_steps) / sum(tests_executed).
  • Every new property is a count. Docs row in docs/telemetry.mdx and changeset included.

Verified

Ran it locally: yes

  • Packed e2e and @e2e-dev/web into a scratch project in /tmp with one deterministic test and one agent.act test, ran e2e run with E2E_TELEMETRY_DEBUG=1 against a local page.
tests_executed: 2, tests_passed: 1, tests_failed: 1,
steps_agent: 1, steps_agent_act: 1, steps_agent_assert: 0, steps_agent_extract: 0, steps_agent_waitfor: 0,
tests_with_agent_steps: 1, tests_with_act: 1, agent_actions: {tap: 2}
  • The act test failed on purpose (the page's button does nothing); the counts are what matter.