feat(e2e): keep artifact screenshots after a secret fill

Merged
OK
8d ago
+119-148

After a test fills a secret, the runner used to stop taking any screenshot for the rest of the attempt. Screenshots are the project's own artifacts, on disk and in CI, so this PR keeps them. Pixels sent to a model stay withheld.

  • app.screenshot(), toHaveScreenshot(), agent.assert evidence, and failure screenshots now work after a fill. Before, they failed with POLICY_DENIED or were dropped.
  • Model-facing pixels don't change: vision, the pixel tools, ctx.pixelsTainted, and MCP screenshot still answer PIXEL_TAINTED after a fill, because model input leaves the machine.
  • SecretExposure.withholdsPixels is renamed to withholdsModelPixels (packages/e2e/src/run/secrecy.ts), so every remaining consumer reads as model-facing.
  • toHaveScreenshot() keeps its other guard: it still refuses when the engine can't prove every secure field on screen is masked.
  • Trade-off: an artifact screenshot masks secure fields only. A secret the app shows anywhere else is in the image, the same as with video, which was never gated. docs/reference/security.mdx says so in a Warning.
flowchart LR
  F[secret fill] --> T{pixel consumer}
  T -- model: vision, pixel tools, MCP --> W[withheld, PIXEL_TAINTED]
  T -- artifact: app.screenshot, toHaveScreenshot, evidence --> K[kept, secure fields masked]

The stacked PR adds agent.act({ onAction }). A per-action screenshot hook is useless if it stops working at sign-in.

Verified

Ran it locally: yes

  • testbed, live model (e2e run --config e2e.agent.config.ts): an agent.act sign-in that fills admin.password through typeSecret, app.screenshot() after each action, then agent.assert. Every screenshot was taken. The password field is masked, and grep -r admin-pass .e2e/ finds nothing.
after the secret fillafter Sign in
after fillafter sign in