How a platform trial is run and recorded
Part of the epic "Epic: slice 0, foundations. The repo holds the line, its context is tested, and the platform facts are recorded". A trial puts one question of §6 to the real GitHub, once, and leaves what it saw as a fixture; the simulator's fake GitHub is held to those fixtures (§4.5). The form has to exist before the first trial, "Trial P-3: an environment secret is unreadable from a work-item branch".
What this story holds:
- Trials run in combsmith/software-factory-canary. Nothing here reads it at run time: only a document, a fixture, its example and its schema name it, and so does the Apps' installation data (
config/apps.json, its schema and its counter-examples), which declares where the Apps are installed and which nothing runs. A fixture arrives by pull request. - A fixture has a schema,
schemas/trial.schema.json, a golden example, and a counter-example that ran outside the canary. - The procedure is the skill
.claude/skills/run-a-platform-trial/SKILL.md, and the rootAGENTS.mdpoints to it. - Each rule the fake GitHub applies is listed in
sim/grounding.pywith the trials that ask it and its fixture, or none.make verifyprints the unverified ones: today all 22.
| Part | Holds |
|---|---|
schemas/trial.schema.json | trial, question, pass, repo (the canary only), ran_at, exchanges of request and response, removed, verdict |
tools/redact_trial.py | Takes a raw capture, JSON or what gh api --include writes, outside the repo. Deletes every field named for a credential, every header not on the list a fixture may keep, and every string or line shaped like a credential. Names where, never what |
tools/check_trials.py | Every file in trials/ is a valid fixture named for its trial. Credential shapes in any case and anywhere in a string: GitHub's tokens, GHSAT, sk-ant-, auth schemes, signed tokens, userinfo and signed URLs, set-cookie or authorization inside text, a 40-hex value beside an auth word, a runner's registration token or JIT config, and private keys, plainly or in base64. A key is refused if, normalised, it names a credential, and a header must be on HEADERS. Examples, counter-examples and trials/AGENTS.md are scanned too. No file names the canary but documents, fixtures, their example and schema, and the Apps' installation data. Each rule in sim/grounding.py is marked in sim/world.py and the reverse, and a named fixture exists, validates and records a trial that asks the rule. Runs alone as python3 -m tools.check_trials |
tools/check_repo.py | A fixture directly in trials/ alone may name a model or the proof of concept's private repo, as what GitHub said |
tools/check_context.py | A skill's frontmatter has its name (quoted or not) and a description (folded or not); the root file points to it by path; the paths it names exist |
tools/verify.py | Prints each unverified rule; it fails nothing |
The skill's rules: capture into mktemp -d and read only what the redaction wrote. Never in the session: a call that mints a credential, gh auth token, --show-token, --verbose, GH_DEBUG, set -x. A trial workflow in the public canary masks or redacts before it prints or uploads anything. GH_REPO is the canary. The batches of §6.1 are followed. The canary is cleaned before the hand-over: tokens revoked, and deploy keys, runners, webhooks, App installations, environments, secrets, branches and pull requests removed. A fail goes to the Maintainer, who opens the issue against the design. No real credential is stored until P-3's fixture has the verdict pass.
What is given up
- The schema accepts the canary alone as
repo. P-8 (twelve repos), P-11 (the control room and the hub), P-19 (a separate public repo), P-30 (the Factory's repo) and P-31 (every installed repo) need more; the skill stops and asks the Maintainer before any of them runs. - The fake keeps one Run in progress and any number queued. GitHub's concurrency group keeps one running and one pending and cancels the older pending one. It is listed as unverified, noted in
sim/AGENTS.md, and left as it is. - Eight rules have no trial in §6, and can be grounded only once one is added there.
- Not caught by shape: a bare 40-hex value (a commit hash looks the same), a token split across two strings, and a one-time code alone. The key and header lists guard those.
- The forbidden commands are a written rule; a hook may hold them later, in "Session hooks that refuse: protected paths, merging, and the repo's own rules".
- Fixtures are exempted, not redacted, for a model's name: it is what was observed.
- The fake does not yet answer in GitHub's shapes, so a fixture is cited and not compared. That waits on the adapters.
- The canary check and the scan of the fixture's examples walk the tree through
tools/walk.py, as "Verify passes in a checkout that holds other worktrees" (#71) requires. So they skip another working tree, known by its.git, and the corpus at the root, as every other check does. - Merging with "Session hooks that check: the fast checks after an edit, verify before a stop, the Maintainer's identity" (#66):
tools/verify.pychanges in both. Afterwards the new test callsverify.main([]).base64is listed intools/check_repo.pyon a line of its own, apart from the one #66 edits.
Verification
make verify: 168 tests OK (2 expected failures), every check ok, "fake GitHub 22 unverified".python3 -m tools.mutants: 33 of 33 mutants killed.tests/test_tools.py, 14 new tests. The security pass's probes are table rows: every credential shape is refused nested in a list of lists, and ordinary GitHub values and keys are kept. The redaction is fed a drafted fixture, a raw capture and plain text; what comes out passes the check, keeps the rest, and the output never repeats a credential. Front-door tests cover the skill check and the canary check, including the Apps' installation data, a nested working tree and the corpus. Grounding is refused for a missing fixture, an invalid one, one of another trial, and one for a rule no trial asks.- Each new rule was broken by hand in a scratch copy, 36 breaks. A test caught every one.
- Run on an export of the commit.
Closes #9
🤖 Generated with Claude Code
Summary by CodeRabbit
-
Documentation
- Added guidance for running platform trials, preparing validated fixtures, and documenting simulator assumptions.
- Added definitions and requirements for trial records and their use in grounding simulator behavior.
-
Tools
- Added checks for trial fixture validity, credential exposure, and consistency between fixtures and documented simulator rules.
- Added a tool to redact sensitive content from captured trial data before it is used as a fixture.
- Verification now reports simulator rules that are not yet supported by a trial fixture.