feat(e2e): keep artifact screenshots after a secret fill
Merged
OK
8d ago+119+119+012345678901234567890123456789-148-148-012345678901234567890123456789
After a test fills a secret, the runner used to stop taking any screenshot for the rest of the attempt. Screenshots are the project's own artifacts, on disk and in CI, so this PR keeps them. Pixels sent to a model stay withheld.
app.screenshot(),toHaveScreenshot(),agent.assertevidence, and failure screenshots now work after a fill. Before, they failed withPOLICY_DENIEDor were dropped.- Model-facing pixels don't change: vision, the pixel tools,
ctx.pixelsTainted, and MCPscreenshotstill answerPIXEL_TAINTEDafter a fill, because model input leaves the machine. SecretExposure.withholdsPixelsis renamed towithholdsModelPixels(packages/e2e/src/run/secrecy.ts), so every remaining consumer reads as model-facing.toHaveScreenshot()keeps its other guard: it still refuses when the engine can't prove every secure field on screen is masked.- Trade-off: an artifact screenshot masks secure fields only. A secret the app shows anywhere else is in the image, the same as with video, which was never gated.
docs/reference/security.mdxsays so in aWarning.
flowchart LR
F[secret fill] --> T{pixel consumer}
T -- model: vision, pixel tools, MCP --> W[withheld, PIXEL_TAINTED]
T -- artifact: app.screenshot, toHaveScreenshot, evidence --> K[kept, secure fields masked]
The stacked PR adds agent.act({ onAction }). A per-action screenshot hook is useless if it stops working at sign-in.
Verified
Ran it locally: yes
- testbed, live model (
e2e run --config e2e.agent.config.ts): anagent.actsign-in that fillsadmin.passwordthroughtypeSecret,app.screenshot()after each action, thenagent.assert. Every screenshot was taken. The password field is masked, andgrep -r admin-pass .e2e/finds nothing.
| after the secret fill | after Sign in |
|---|---|