test(authoring): evaluate lower-test context in E2E authoring

Draft
KN
7d ago
+6932-9

Why

  • Measure whether inspecting lower-level tests improves E2E authoring before changing production guidance.
  • The forty-run pilot reached its provisional detection ceiling. Six connected product tickets check whether a harder baseline leaves useful headroom.

What changed

  • Add two regressions per ticket, correct-only budget calibration, immutable first attempts, and separate human attribution.
  • Publish the complete twelve-session baseline capture, including every unusable execution. The result remains unresolved-operational-feasibility.
  • Fix worker retirement to use the configured disposal grace. Isolate the failed-write and synthetic timeout controls from unrelated setup operations.
flowchart LR
  A[Freeze and verify the bank] --> B[Six correct-only calibration authors]
  B --> C{All six usable?}
  C -->|No| D[Retain failures and stop admission]
  C -->|Yes| E[Seal common budget]
  E --> F[Twelve baseline authors and 72 fixed cells]
  F --> G[Validate executions and human attribution]

Scope

  • Keep this PR draft and production authoring guidance unchanged. No sixty-run comparison ran or is authorized by these results.
  • All twelve authors finished. Two correct-app repeats are unusable, and twenty member attributions await human judgment. All forty-eight changed-app repeats failed, which supplies no passing candidate for qualifying headroom but does not establish intended detection.
  • Raw captures and the external operator remain private. Public aggregates do not independently reproduce the live capture.
  • Maintainer review of the study scope and corpus is the next decision before another cohort. Human labels cannot repair an unusable execution.

Blast radius

  • The study is a private repository app. The scheduler correction also affects ordinary worker retirement and final disposal. It reuses the configured grace and changes no public signature.

Verified

Ran it locally: yes

  • Copied built CLI and real Chromium. node apps/test-authoring/connected-cli.ts verify-bank --experiment BANK verified 36 lower launches and 108 browser controls. Fresh independent review accepted the exact bank and native MCP admission before calibration.
  • Six validate-calibration calls passed both correct-app repeats for all six exact exports. seal-baseline froze 46 completed responses, 2,100,000 full reported tokens, and 720 authoring seconds. All twelve baseline first attempts completed within those limits.
  • Twelve grade calls retained all 72 predefined execution cells. node apps/test-authoring/connected-cli.ts summarize --experiment BANK retained ordered gate 2, unresolved-operational-feasibility. The final summary retains two unusable slots and ten pending-human slots. No failed cell was retried or repaired.
  • The complete baseline result records raw outcomes, validity failures, calibration sources, and limits. Earlier pilots remain separate. No inspection-benefit conclusion follows.
  • Earlier built-CLI disposal verification remains recorded in the first connected report.
mainthis branch
A fixed ten-second retirement timer kills the worker before its configured thirty-second engine cleanup budget ends.A 12.34-second disposal pause completes without cleanup error or remaining process. A pause beyond the engine budget produces CLEANUP_TIMEOUT, cleanup phase, and CLI exit 3.