Trial scripts: P-13's script hardened before it runs again
Why
"Trial scripts: P-13's script hardened before it runs again" (issue 178). tools/trial_p13_cli.sh is kept as it ran for trials/P-13.json, and the review of "Trial P-13 (pass): the model CLI returns validated structured output and usage, headless, at 2.1.302" (pull request 177) found what it should do better before it, or a script copied from it for P-26 or a metered run, runs again. Above all it removed the model's text by naming what to drop, so a field a new CLI version adds would pass through.
What changed
- The Task's answer is kept by allowlist. Only the fields
KEPT_RESULTnames reach OUT:type,subtype,is_error,num_turns,duration_ms,duration_api_ms,total_cost_usd,usage,modelUsage,stop_reason,permission_denials(tool names alone) andstructured_outputonce it validates. Every other field is dropped and counted inchecked.json. Within the kept fields, at any depth, a string is kept only under a keySTRING_KEYSnames and only in a code's shape (letters, digits,_ . : -, at most 64), and a key is held to that shape too, else dropped (a key with what is under it) and counted: the CLI answers in numbers and codes under known keys, and nothing else looks like that. - CONFIG_DIR is refused anywhere under
~/.claudeand at the home directory or any directory holding it (so/too), not only at~,~/.claudeand/; the paths are compared as the file system resolves them and without case, since macOS does not tell.claudefrom.CLAUDE. - The login keeps a kept field's value only as a bool or a code, so an address or prose under a named field is dropped, and stops the script before the Task unless
auth statusexits 0 and says logged in with a true bool: a non-zero exit or a status the script cannot read no longer goes on. Exchange 2's exit code is recorded inchecked.jsonaslogin.exit, with its status, on the early stop as well;code2was set and never used. - The model runs with no tool:
--tools "", which the CLI's own help describes as disabling all tools. The stand-in test pins the option; the next real run is the first with it, and shows that structured output still works that way. This is the one box the Maintainer's run ticks. - Tests run the script against a stand-in CLI that records the environment and arguments it is given: exactly the named environment and no key from the shell (the test's shell holds a fake
ANTHROPIC_API_KEYandGH_TOKEN); each refusal (OUT exists, inside the repo, inside CONFIG_DIR; CONFIG_DIR inside the repo, the home directory, a directory holding it,/,~/.claude, under it, linked to it,~/.CLAUDE, missing); the wrong version, a login that says no, an unknown status, a status that is not JSON and anauth statusthat exits non-zero, each before the Task is asked; and that no planted model text reaches OUT, whether the structured output validates, does not validate, comes inresultalone, is missing, or the answer is not JSON. - The runbook moves into the repo as
tools/trial_p13_runbook.md, beside the script, without the two-separate-logins check (decision 0033), and without its blocked-until section and open questions, which that decision and this issue answered. The header andrequests.jsonsay a configuration directory of its own, logged in to the Maintainer's own subscription. - SonarCloud's 16 S7688 notes: every
[is[[. KEY_HARMLESSnames the CLI's camelCase and nested counts (inputTokens,outputTokens,cacheReadInputTokens,cacheCreationInputTokens,maxOutputTokens,thinkingTokens,thinking_tokens,ephemeral_1h_input_tokens,ephemeral_5m_input_tokens,output_tokens_details), compared in lower case as the check does, so a fixture keeps them; the credential-key test now tries each as kept and still refuses a credential's key.trials/P-13.jsonis unchanged: its counts were removed at capture and are not recoverable.- A key shaped like a credential is a credential to
tools/check_trials.pyandtools/redact_trial.py, and its place is named without it anywhere a message names a place: a field's path, a header's, a schema's error, the text redaction's report.api_key,apikey,private_key,privatekey,passphrase,authentication,passwd,passcode,bearer,access_key,accesskey,encryption_key,signing_keyandmaster_keyname a credential now, and every message of the check hides a credential's shape, an escaped separator read as the separator. - The Task's standard error is counted, not kept: it may hold the model's text, so
checked.jsonrecords its line and character counts and noclean-3.stderrreaches OUT. - Rule rows on
trials/AGENTS.mdand on decision 0033's Boundaries; thirty-nine mutants, one per rule, in the script,check_trials.pyandredact_trial.py.
Measured
make verify passed on a git archive export of each head through 4393742, run alone. At the last two heads the machine carried a load of 6 to 8 from other sessions, and the only failures were the two hook tests bound to a 5-second wall clock, at 5.3 and 5.6 seconds, which this change does not touch and the mutation check skips; every other test passed, and CI's checks job is the word on those two. The thirty-nine mutants were killed by hand on scratch copies with their named tests. The script was not run against the real CLI: the stand-in is a shell script that answers from files and writes what it was given beside itself.
Who read it
Muse Wonder Full read this branch seven times, before the Maintainer's ruling of 2026-10-20 that Muse reads Kernel and hook pull requests only; the reads are recorded below as they happened. Claude's review and security pass, CodeRabbit, Codex (two reads before its limit) and SonarCloud read it as well.
Muse Wonder Full read the branch before the push, at f5687c0, and found three holes, each fixed in the one round: the allowlist held only top-level names, so prose planted in stop_reason, total_cost_usd or under usage reached OUT with every part holding (every kept value is now held to numbers, codes and short words); the ~/.claude refusal compared case, which macOS's file system does not (paths are compared without case now); and a CONFIG_DIR linked to ~/.claude was refused by pwd -P alone, with no test or mutant holding it (both added). Its read of that fix round, at 1a48a1b, found four more, each fixed in the second round: keys under kept fields were never filtered, so prose or a credential-shaped key passed; the prose test looked for a space alone, so tabs passed; the test took the kept list from the script itself, so widening it was held by nothing; and thinking_tokens under output_tokens_details was still refused. Its read of the second round found seven more, three real and four coverage gaps, each fixed in the third round: a header's path, a schema's error and the text redaction's report named a credential-shaped key; api_key-style names were not a credential's; the Task's standard error was kept with only the credential shapes removed; and the long word, the version exit code and OUT through a link had no case or mutant. Its read of the third round found three more, fixed in the fourth: a credential whose shape spans a space passed the per-word hiding of a schema's error (hidden whole now); a key with a dot in it could borrow a named suffix (a string is kept by its own key now); and the login's values were kept by name alone, so loggedIn: 1 read as yes and an address could sit under a named field (a bool or a code only, and only a true bool is logged in). Its read of the fourth round found five more, fixed in the fifth: the message for a fixture that cannot be read quoted its duplicate keys; a schema's error was tested after its repr escaped the separator; authentication, passwd, bearer, access_key and the like were not a credential's names; a list under a named key inherited the key's trust; and HOME through a link had no case or mutant. Its read of the fifth round found four more, fixed in the sixth: a repr's \x0b or \xa0 was not read as a separator (every escape is one now); ~/.claude itself a link was compared unresolved; a version that moved under the run still held, since the stand-in never changed its answer (it can now); and auth_key and pwd were not a credential's names. Its seventh read, the last by the Maintainer's ruling of 2026-10-20, read the tree before the two weak cases were strengthened and named those two survivors first; of the rest, two were real holes in the hiding this change adds and are fixed (encryptionKey, signingKey and masterKey were refused only in snake case; a raw control character in a key split a credential's shape, so the place was named by the credential), and two minors, with CodeRabbit's reason pattern, are recorded in "Trial scripts: what Muse's later rounds asked of trial_p13_cli.sh" (issue 247) for a change of their own. The filtered mutation run of the script alone, the one local run allowed, killed every mutant of the script with no FLAKY verdict after the test's clean-up, and showed one mutant of mine to be a rule the bool filter already held, which is gone. Two CI shards failed on the third round for a reason of the test's own: under the mutant that reads OUT as written, the script made OUT through the link inside the runner's working copy, which is lent to the next mutant, so the unbroken rerun found it existing and refused for the wrong reason; the test now names those directories itself and removes them after each case. CodeRabbit read 4393742 and found the record wrong: the walk over the kept answer ran while the structured output was still in it, so a passing run's checked.json said the Result's own strings were taken out while clean-3 kept them; the Result and the refused tools are set aside before the walk now, and the test asserts the record names nothing kept. CodeRabbit read eb294f5 and caught a regression of the last round: reading a control character as a separator turned a trailing newline into a space, and the registration-token shape, anchored at the end, had matched before a final newline; it matches the stripped text now, with a case and a mutant. Codex read 1a48a1b and found two more: a one-word string under a key the CLI adds passed as a code (now a string needs a named key as well as the shape), and on Linux's bash the ancestry pattern let CONFIG_DIR=/ through, which the CI run of that head showed as the one failing test (the trailing slash is kept now, and the case fails on the runner as it should). The Claude review found the first counts filter evaluated each value twice, so a dropped string was counted twice and a legitimate null was dropped; it was rewritten with a sentinel before the push, with the null and a short word in the test. The security pass found no defect: the script gives the CLI less than before (no tool, an allowlist in place of a denylist) and the tests prove no key from the shell reaches it.
Closes #178.
🤖 Generated with Claude Code