Trial P-51 (fail): with the subprocess scrub on, the pinned CLI's sandbox lets every write outside the work directory land

Merged
DO
7d ago
+1728-6

Closes #312. The open design question it leaves, Intents that need a tool, is #314.

Trial P-51: fail

Question. On a GitHub-hosted Linux runner, does the pinned model CLI's sandbox let a tool's write outside its work directory land, and is it the subprocess scrub or the order of the Tasks that decides?

Pass condition, written at 2026-10-22T12:12:39Z, before anything ran, and held at the top of tools/trial_p51_sandbox_writes.sh: in every configuration a Run can be given (the scrub unset, 0 or 1), in both orders, at the pinned version before and after:

  • the CLI exited 0 and its answer is no error;
  • the tool's result went back and is not an error;
  • the probe ran;
  • none of the four writes outside the work directory landed.

How it ran. As P-50 did: in the canary, a workflow with no secret, only contents: read on its job, and ubuntu-24.04 with kernel.apparmor_restrict_unprivileged_userns=0 (P-50 showed the tool does not run without it). The CLI was at 2.1.306, and its API was P-50's stub on 127.0.0.1, with a dummy key and no model. There were three configurations: CLAUDE_CODE_SUBPROCESS_ENV_SCRUB unset, 0 and 1, with nothing else changed between them. Each Task had its own work and configuration directories, and the three ran in one order, then in reverse. The probe tried four writes outside its work directory:

  • a file in the home directory;
  • a dotfile there;
  • a file in /tmp;
  • a file in the work directory's parent, which was under the runner's own temporary directory, not /tmp.

It also wrote one file inside the work directory, as a control.

The fixture holds the trial's run, 39781351303, and the run before it, 39778822341, which showed the same in every configuration.

What it showed, in both orders:

Scrubhomedotfile/tmpparent
unset (GITHUB_ACTIONS set, the runner's other variables removed)refusedrefusedrefusedrefused
0refusedrefusedrefusedrefused
1landedlandedlandedlanded

In every Task, the tool ran in a PID namespace of its own, its result went back with no error, and the control write landed.

So on Linux at 2.1.306, the scrub that takes the credential's listed name out of a tool's environment also opens the sandbox to writes outside the work directory. The ruling is on #194: no step sets the scrub unless it also passes --tools "", and no step holding the model credential has any tool. Intents that need a tool are #314.

What this changes

  • tools/trial_p51_sandbox_writes.sh, byte for byte as the canary ran it. With it come its stand-in tests (SandboxWritesTrial, run against P-50's real stub), six mutants, and a test that reads the fixture's analysis again from the job's log.
  • trials/P-51.json.
  • Design §4.9: the sentence on writes, and "until P-26, P-50 and P-51 pass". §6: a new P-51 row. §6.1: P-51 is listed beside P-26 and P-50.
  • trials/AGENTS.md: P-51's rules.

The script fixes the faults P-50's had, listed on #312: it reads the stub's port through a pipe, gives each Task its own configuration directory, and checks that the tool's result is not an error. The walk faults do not apply, since P-51 does not walk processes.

Canary

The branch trial/p-51 is deleted after this PR opens. Its runs stay in the canary, and their logs hold no secret: the key is a dummy, and the printed records are booleans, counts and exit codes. Every file the probe wrote outside its work directory was removed by the script in the same job. No token was minted, and nothing else was made.

Gates

  • PATH=/usr/bin:/bin SIM_SEEDS=40 make verify: 1013 tests, OK, at c0958c1, before main merged in (the merge brought only #302's and #303's changes, which touch none of this)
  • Claude review with a security pass: no blockers or majors. Its minors on cleanup and wording are fixed, which needed one more canary run, and the result was the same. The suggestion to require the control write in a future pass condition is noted for whoever runs this again.
  • P-51's six mutants, applied by hand: all killed.
  • Muse: not run (Maintainer, 2026-10-20: Kernel/hook only).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Clarified the documented sandbox trial results across three subprocess-scrub configurations, including which attempted writes were blocked and when external writes occurred.
    • Updated the trial order and runbook guidance to include the new sandbox-write trial.

  • Tests

    • Added validation for trial results, configuration isolation, and cleanup of files created outside the trial work directory.