feat(agent)!: keep model screenshots after a secret fill
Merged
OK
18d ago+461+461+012345678901234567890123456789-879-879-012345678901234567890123456789
What and why
- A secret fill no longer turns off model screenshots for the rest of the attempt.
vision: true,vision: 'only', thescreenshotand point tools,e2e mcp, ande2e explorefindings keep working after a sign-in, in serial groups, and in tests that restore a saved session. - Masking and redaction don't change. Screenshots still mask secure fields, unproven masking is still
MASKING_UNPROVEN, and secret values are still<secret:name>in all text. - Why drop it:
- It blocked every vision step after login, which is where most of a suite's tests run. Mobile needs pixels more than web does, because its trees are weaker.
- It guarded the model provider, the one destination the user picked and that already sees the whole UI. Meanwhile the same value goes unmasked into video and report screenshots (#1219), and to any origin.
- It pushed users to pass secrets as plain strings, which costs them the runner's checks on agent fills.
- Proving that a full frame is redacted isn't possible without OCR or a similar heuristic (see the #1385 review), so a scoped opt-in had nothing sound behind it.
- Accepted risk, documented in
docs/security.mdxanddocs/reference/security.mdx#screenshots-after-a-secret-fill: the model provider now receives a generic secret filled into a plain input, or a value the app shows outside a secure field (for example behind a show-password toggle). The docs say to use test credentials. - Breaking (
e2eminor):StepExecutorContext.pixelsTaintedis removed. An older@e2e-dev/decisionreads it asundefinedand degrades safely.- New steps never report
visionDegraded: "PIXEL_TAINTED". The report schema and theReportSteptype keep the value, because--last-failedcarries old results into new reports.
- Removed internals:
SecretExposureinrun/secrecy.ts.- The taint flag in session envelopes (
schema/session-v1.schema.json). Sessions are per-run, so no old envelope gets loaded. - The taint branch and the pass-through
fallback/apiparams ofretryingObserve. ScreenPresenter'spixelsUnavailable.- The MCP recording's "a secret was filled" line.
start_recordingalready says the video is unmasked.
AGENTS.md,SECURITY.md, about 20 docs pages, andskills/e2e/are updated to the new rule.
await agent.act('sign in', { params: { password: credentials.user('test').password } });
await agent.assert('the landing screen is shown', { vision: 'only' }); // was POLICY_DENIED (PIXEL_TAINTED)
Issue
Fixes #1383. Supersedes the approach in #1385.
Verified
Ran it locally: yes
- I ran the testbed with the built CLI and a live model (
gateway/openai/gpt-6-luna-fast) on a scratch test, not committed:e2e run tests-agent/verify-taint.e2e.ts --config e2e.agent.config.ts --no-cache --video=on.- Test 1 signs in with
fill(credentials.user('admin').password), then callsagent.assert(..., { vision: 'only' }). - Test 2 signs in through
agent.actwith a password param, which meanstype_secret, then makes the same assert.
- Test 1 signs in with
- After the run,
grep -r admin-pass apps/testbed/.e2efinds nothing.
main (origin/main built in a worktree) | this branch |
|---|---|
| 2 failed, 0 judge calls | 2 passed, 1 judge call each, visionOnly and visionInput: true |
main:
FAIL a test fill, then a vision-only assert
POLICY_DENIED: agent.assert was called with vision: 'only', so the screenshot is its only evidence, but pixel evidence is unavailable (PIXEL_TAINTED); it will not answer from the semantic tree instead
FAIL an agent sign-in with a secret, then a vision-only assert
POLICY_DENIED: ... (PIXEL_TAINTED) ...
this branch:
✓ a test fill, then a vision-only assert > agent.assert "the page welcomes the admin back" 2.55s · 1 model call
✓ an agent sign-in with a secret, then a vision-only assert > agent.act "sign in with the given username and password" 8.98s · 3 model calls
✓ an agent sign-in with a secret, then a vision-only assert > agent.assert "the page welcomes the admin back" 2.44s · 1 model call
judge: "The screenshot shows “Welcome back, admin!” on the Dashboard page."
The assertion screenshot after the agent's type_secret sign-in:
- Tests that asserted the taint now assert the opposite:
- pixels after a fill:
agent-vision,agent-fallback-handoff,serial-secrets,session-secrets,explore,mcp-server,mcp-run-code,agent-act-pixel-tools, mobiletool-pixels; - the rejected-fill case in
agent-policynow checks that an engine error echoing the value is redacted.
- pixels after a fill:
- A fresh-context review found these, and I fixed them: stale docs and
SECURITY.mdlines, the report type for carriedPIXEL_TAINTEDresults, the deadfallbackparam, and lost coverage for the rejected fill. - I dropped one review suggestion on purpose: bringing back a fill flag only for the MCP recording warning.
https://github.com/user-attachments/assets/908ca13f-8741-4829-a512-ac3baeca1bc9