feat(agent)!: keep model screenshots after a secret fill

Merged
OK
18d ago
+461-879

What and why

  • A secret fill no longer turns off model screenshots for the rest of the attempt. vision: true, vision: 'only', the screenshot and point tools, e2e mcp, and e2e explore findings keep working after a sign-in, in serial groups, and in tests that restore a saved session.
  • Masking and redaction don't change. Screenshots still mask secure fields, unproven masking is still MASKING_UNPROVEN, and secret values are still <secret:name> in all text.
  • Why drop it:
    • It blocked every vision step after login, which is where most of a suite's tests run. Mobile needs pixels more than web does, because its trees are weaker.
    • It guarded the model provider, the one destination the user picked and that already sees the whole UI. Meanwhile the same value goes unmasked into video and report screenshots (#1219), and to any origin.
    • It pushed users to pass secrets as plain strings, which costs them the runner's checks on agent fills.
    • Proving that a full frame is redacted isn't possible without OCR or a similar heuristic (see the #1385 review), so a scoped opt-in had nothing sound behind it.
  • Accepted risk, documented in docs/security.mdx and docs/reference/security.mdx#screenshots-after-a-secret-fill: the model provider now receives a generic secret filled into a plain input, or a value the app shows outside a secure field (for example behind a show-password toggle). The docs say to use test credentials.
  • Breaking (e2e minor):
    • StepExecutorContext.pixelsTainted is removed. An older @e2e-dev/decision reads it as undefined and degrades safely.
    • New steps never report visionDegraded: "PIXEL_TAINTED". The report schema and the ReportStep type keep the value, because --last-failed carries old results into new reports.
  • Removed internals:
    • SecretExposure in run/secrecy.ts.
    • The taint flag in session envelopes (schema/session-v1.schema.json). Sessions are per-run, so no old envelope gets loaded.
    • The taint branch and the pass-through fallback/api params of retryingObserve.
    • ScreenPresenter's pixelsUnavailable.
    • The MCP recording's "a secret was filled" line. start_recording already says the video is unmasked.
  • AGENTS.md, SECURITY.md, about 20 docs pages, and skills/e2e/ are updated to the new rule.
await agent.act('sign in', { params: { password: credentials.user('test').password } });
await agent.assert('the landing screen is shown', { vision: 'only' }); // was POLICY_DENIED (PIXEL_TAINTED)

Issue

Fixes #1383. Supersedes the approach in #1385.

Verified

Ran it locally: yes

  • I ran the testbed with the built CLI and a live model (gateway/openai/gpt-6-luna-fast) on a scratch test, not committed: e2e run tests-agent/verify-taint.e2e.ts --config e2e.agent.config.ts --no-cache --video=on.
    • Test 1 signs in with fill(credentials.user('admin').password), then calls agent.assert(..., { vision: 'only' }).
    • Test 2 signs in through agent.act with a password param, which means type_secret, then makes the same assert.
  • After the run, grep -r admin-pass apps/testbed/.e2e finds nothing.
main (origin/main built in a worktree)this branch
2 failed, 0 judge calls2 passed, 1 judge call each, visionOnly and visionInput: true
main:
 FAIL  a test fill, then a vision-only assert
POLICY_DENIED: agent.assert was called with vision: 'only', so the screenshot is its only evidence, but pixel evidence is unavailable (PIXEL_TAINTED); it will not answer from the semantic tree instead
 FAIL  an agent sign-in with a secret, then a vision-only assert
POLICY_DENIED: ... (PIXEL_TAINTED) ...

this branch:
   ✓ a test fill, then a vision-only assert > agent.assert "the page welcomes the admin back" 2.55s · 1 model call
   ✓ an agent sign-in with a secret, then a vision-only assert > agent.act "sign in with the given username and password" 8.98s · 3 model calls
   ✓ an agent sign-in with a secret, then a vision-only assert > agent.assert "the page welcomes the admin back" 2.44s · 1 model call
judge: "The screenshot shows “Welcome back, admin!” on the Dashboard page."

The assertion screenshot after the agent's type_secret sign-in:

assert screenshot after secret sign-in

  • Tests that asserted the taint now assert the opposite:
    • pixels after a fill: agent-vision, agent-fallback-handoff, serial-secrets, session-secrets, explore, mcp-server, mcp-run-code, agent-act-pixel-tools, mobile tool-pixels;
    • the rejected-fill case in agent-policy now checks that an engine error echoing the value is redacted.
  • A fresh-context review found these, and I fixed them: stale docs and SECURITY.md lines, the report type for carried PIXEL_TAINTED results, the dead fallback param, and lost coverage for the rejected fill.
  • I dropped one review suggestion on purpose: bringing back a fill flag only for the MCP recording warning.

https://github.com/user-attachments/assets/908ca13f-8741-4829-a512-ac3baeca1bc9