Trial P-39 (pass): each App's grants are read from GitHub and the data is held to them; each key lives in the environment of the one job that holds it

Merged
DO
1mo ago
+3434-30

Closes #26.

The trial

P-39, a new row in §6, batch 1: "Does what GitHub granted each Factory App, its permissions, its events and the repos it reaches, agree with config/apps.json?" Pass, as ruling 3 states it: for each App, app_id is the data's; the permissions and their levels are exactly the data's, no more and no fewer, and the same registered and installed; events is empty; the selection is selected; and the repos are exactly installed_on.

Verdict: pass, for all three Apps, on two fixtures:

  • trials/P-39.grants.json (2026-10-18): GET /apps/{slug} for each App, and GET /orgs/combsmith/installations, kept whole, read through the Maintainer's login. Nothing was minted.
  • trials/P-39.reach.json (2026-10-19T03:24:52Z): GET /installation/repositories for each installation. The Maintainer ran tools/trial_p39_reach.sh at the Maintainer's own terminal. Each token was narrowed to metadata: read, used once and revoked, and each revocation returned 204. The session built the fixture from the script's output directory alone.
Appapp_idInstallationSelectionReaches
combsmith-sf-controller5364130172571513selectedsoftware-factory, software-factory-canary
combsmith-sf-pr-review5364144172571064selectedsoftware-factory, software-factory-canary
combsmith-sf-reader5364151172572220selectedsoftware-factory, software-factory-canary

Each list fits in one page (no link header). Oversight posted its verdict on #26.

What was removed

  • P-39.grants.json: each request's authorization header, and each installation's access_tokens_url in the installations list (17 entries).
  • P-39.reach.json: each request's authorization header. The redaction found no credential in the response. It also dropped ten response headers per exchange that a fixture may not keep (x-github-api-version-selected, access-control-*, strict-transport-security, x-frame-options, x-content-type-options, x-xss-protection, referrer-policy, content-security-policy, x-github-edge-region). Like the grants fixture, removed lists only where credentials were. A second redaction pass over the drafted fixture removed nothing.

Public exposure

trials/P-39.grants.json names every App installed on the org, third-party Apps included, with their permissions. Its x-oauth-scopes header shows the scopes of the Maintainer's gh login. Ruling 2 accepted keeping the installations read whole, and six other fixtures already carry that header.

Nothing was made in the canary, so step 8 had nothing to undo. No rule of the fake asks P-39, so nothing is grounded in sim/grounding.py.

The environments and keys (ruling 6)

All are on combsmith/software-factory. Only names, dates and branch policies were read.

EnvironmentSecretCreatedDeployment branchesAdmin bypassRequired reviewer
factory-pollerCONTROLLER_APP_KEY2026-10-19T03:39:48Zmain (branch)offnone
factory-applyPR_REVIEW_APP_KEY2026-10-19T03:39:58Zmain (branch)offnone
factory-runREADER_APP_KEY2026-10-19T03:40:06Zmain (branch)offnone

Each environment holds one secret. There is no repository-level secret. The Maintainer stored each key himself, and only after its App passed.

What changes

  • config/apps.json: each App gets its app_id and a key of {environment, secret}, given by name, never by value. Schema: schemas/apps.schema.json.
  • tools/check_apps.py: holds each App's permissions and levels to what P-39 recorded for the installation of its app_id, and checks that installation belongs to the App so named. With no such record, it trusts nothing. It holds each App's key to the environment of the job that holds it, by a fixed map: the Controller's to factory-poller, the reader's to factory-run, and every Role's to factory-apply. Several Roles' keys may share factory-apply, as §4.9 gives apply each Role's App key. Each secret is named for its App. Swapping the Controller's and the reader's keys is refused. There are six new mutants, and the tests in tests/test_tools.py::Apps kill them.
  • schemas/trial.schema.json: repo is the canary, the org combsmith (a trial about the org and its Apps), or null (a vendor's CLI, from P-13). The counter-example is examples/invalid/trial.neither-the-canary-nor-the-org.json. tools/check_trials.py pairs the org with what was asked, as it pairs null with a subscription: repo is combsmith exactly when every call is of an App (/apps/…), the org (/orgs/combsmith/…) or an installation (/installation/…). The test covers both directions, and two mutants back it.
  • read:facts.queue now waits on #185 (P-2, the merge queue), per ruling 7, and no longer on this issue.
  • tools/trial_p39_reach.sh, changed after the run: OUT is resolved, with links and .. followed, before it is held to be outside the repo, and a revocation that does not return 204 warns that the token may be live for up to an hour and must be revoked by hand. The fixture records the run, not this guard.
  • §6 adds the P-39 row with its verdict, and §6.1 puts P-39 in batch 1. The skill and trials/AGENTS.md say the same.
  • Decision record 0035: "An App's grants are read from GitHub and the data is held to them; each key lives in the environment of the one job that holds it." It decides one row of decision 0033 otherwise: a fixture's repo may also be the org, for a trial whose every call is of the org or its Apps.

Protected edits for the Maintainer's review (write 4): §6's P-39 row, decision 0035 and config/apps.json.

Merge order

The record takes the next free number at enqueue, when this PR is clean and green. This PR enqueues before #161, so its record is 0035, the next after main's 0034. #161 takes the next free number when it enqueues. If another record reaches main first, this one is renamed to the next free number before it is enqueued.

Verify

  • make verify passes in full on a clean git archive export, with PATH=/usr/bin:/bin and SIM_SEEDS=40.
  • Mutants were swept by hand on each target file this branch changes (tools/check_apps.py, tools/check_trials.py). Each is killed by a test that passes without it.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Platform trials can now cover organization-level questions and GitHub Apps, in addition to canary-repository trials.
    • App configuration checks now verify that registered permissions and repository access match GitHub, and that each app key is assigned to its designated job environment.
  • Documentation
    • Trial guidance now includes organization and App trials, with updated examples and decision criteria.