test(authoring): evaluate lower-test context in E2E authoring
Draft
KN
7d ago+6932+6932+0123456789012345678901234567890123456789-9-9-0123456789
Why
- Measure whether inspecting lower-level tests improves E2E authoring before changing production guidance.
- The forty-run pilot reached its provisional detection ceiling. Six connected product tickets check whether a harder baseline leaves useful headroom.
What changed
- Add two regressions per ticket, correct-only budget calibration, immutable first attempts, and separate human attribution.
- Publish the complete twelve-session baseline capture, including every unusable execution. The result remains
unresolved-operational-feasibility. - Fix worker retirement to use the configured disposal grace. Isolate the failed-write and synthetic timeout controls from unrelated setup operations.
flowchart LR
A[Freeze and verify the bank] --> B[Six correct-only calibration authors]
B --> C{All six usable?}
C -->|No| D[Retain failures and stop admission]
C -->|Yes| E[Seal common budget]
E --> F[Twelve baseline authors and 72 fixed cells]
F --> G[Validate executions and human attribution]
Scope
- Keep this PR draft and production authoring guidance unchanged. No sixty-run comparison ran or is authorized by these results.
- All twelve authors finished. Two correct-app repeats are unusable, and twenty member attributions await human judgment. All forty-eight changed-app repeats failed, which supplies no passing candidate for qualifying headroom but does not establish intended detection.
- Raw captures and the external operator remain private. Public aggregates do not independently reproduce the live capture.
- Maintainer review of the study scope and corpus is the next decision before another cohort. Human labels cannot repair an unusable execution.
Blast radius
- The study is a private repository app. The scheduler correction also affects ordinary worker retirement and final disposal. It reuses the configured grace and changes no public signature.
Verified
Ran it locally: yes
- Copied built CLI and real Chromium.
node apps/test-authoring/connected-cli.ts verify-bank --experiment BANKverified 36 lower launches and 108 browser controls. Fresh independent review accepted the exact bank and native MCP admission before calibration. - Six
validate-calibrationcalls passed both correct-app repeats for all six exact exports.seal-baselinefroze 46 completed responses, 2,100,000 full reported tokens, and 720 authoring seconds. All twelve baseline first attempts completed within those limits. - Twelve
gradecalls retained all 72 predefined execution cells.node apps/test-authoring/connected-cli.ts summarize --experiment BANKretained ordered gate 2,unresolved-operational-feasibility. The final summary retains two unusable slots and ten pending-human slots. No failed cell was retried or repaired. - The complete baseline result records raw outcomes, validity failures, calibration sources, and limits. Earlier pilots remain separate. No inspection-benefit conclusion follows.
- Earlier built-CLI disposal verification remains recorded in the first connected report.
| main | this branch |
|---|---|
| A fixed ten-second retirement timer kills the worker before its configured thirty-second engine cleanup budget ends. | A 12.34-second disposal pause completes without cleanup error or remaining process. A pause beyond the engine budget produces CLEANUP_TIMEOUT, cleanup phase, and CLI exit 3. |