PR #666 - preloopdev/preloop

What & why

The supply-chain guard was 50 lines of shell inside the workflow, so a contributor could only learn the rule from a failed CI job — and it carried a dependency-migration label exception that let a lockfile ship with source changes once a maintainer applied the label to the head SHA.

This PR moves the logic into supply-chain/guard-policy-files.sh, deletes the exception and the label trigger, and documents the rule where it lives. The script is the only copy of the rule: the engine runs it as the Supply chain audit check, and a contributor runs it with just supply-chain-guard (part 2).

Part 1 of 2, and the half that is still open. Part 2 (#667) merged as 7228f64c (2026-10-17T23:31Z) and put in main: the just supply-chain-guard recipe, the AGENTS.md / CONTRIBUTING.md / PR-template text, the CHANGELOG.md entry, and the retargeted wire-format test (dependency-migration -> dependencies). It added no policy file, exactly as it said.

That is the inverse of this branch: main today has the recipe, the docs and no supply-chain/guard-policy-files.sh, supply-chain/tests/ or supply-chain/README.md, so just supply-chain-guard on main calls a script that does not exist and supply-chain.yml still carries the inline 50-line guard with the dependency-migration exception. This PR is the missing half and the only remaining copy of the rule; it duplicates nothing #667 landed.

main is merged into this branch (e41cd62d6, base 66b68926d); the merge was clean, so the two halves compose with no conflict.

The rule after this PR

Dependency/policy files are deny.toml, .cargo/audit.toml, .github/workflows/supply-chain.yml, supply-chain/**, and any Cargo.toml or Cargo.lock anywhere in the tree (workspace root, crates/*, benchmarks/*, a standalone crate such as packaging/crates-io/preloop, any member added later).

any combination of those, and nothing else -> pass (Dependency/policy PR detected: OK), so the lockfile, the audit policy, the vet state and the manifests can share one PR and one CI run;
none of those -> pass (Policy file guard: OK (no policy or manifest files modified));
one of those mixed with anything else (source, tests, docs, CHANGELOG.md, another workflow, the justfile, a script) -> fail.

Code that needs a new dependency therefore lands in two PRs: the dependency/policy PR first, the code PR rebased on main afterwards.

Changes

supply-chain/guard-policy-files.sh <base-rev> <head-rev> — one classification rule, git diff --name-only $(git merge-base base head) head as the only input. Exit 0 sanctioned, 1 violation, 2 undecidable (bad usage, unresolvable revision, no merge base) so it fails closed. It never fetches: the job keeps the fail-closed git fetch --no-tags origin "$base" "$pr_head".
.github/workflows/supply-chain.yml — the step calls the script; the DEPENDENCY_MIGRATION env expression and the labeled trigger type are gone.
Failure message lists both file sets, names the split, states the sanctioned set, and offers no override. It does not mention the label.
supply-chain/README.md — the rule, the two shapes, the ordering, how to run the guard locally, and what CI does.
supply-chain/tests/guard-policy-files.test.sh — 28 cases over scratch git init repositories: every allowed combination passes, each path mixed with one non-policy file fails, plus the merge-base case (both tips identical, so a tip-to-tip diff is empty and would hide the policy touch) and the three fail-closed paths.
Manifests and lockfiles match at any depth ((^|/)Cargo\.toml$, (^|/)Cargo\.lock$), so benchmarks/preloop-perf, benchmarks/control-load, packaging/crates-io/preloop and any future workspace member have a sanctioned shape instead of an impossible one.

Review follow-ups (this head)

Three threads were real and are fixed here; the rest are answered on their threads.

The guard's own suite had no gate (devin, codex). .github/workflows/supply-chain.yml now runs supply-chain/tests/guard-policy-files.test.sh as the step Guard policy files (self-test), before the guard judges anything, on every trigger. The workflow is the only place it can live: justfile and ci.yml are outside the policy set, so a recipe or a ci.yml step in this PR would make this PR fail its own guard.
Quadratic classification (superagent-security; this is what failed Superagent Security Scan). The classifier rebuilt two growing scalars, so each path copied the whole accumulated list: O(n²) work over an attacker-controlled diff. It now appends to two Bash arrays (amortized O(1)). On 12,000 mixed paths: 27.6s -> 0.80s (bash 5.3) and 27.8s -> 0.84s (macOS /bin/bash 3.2). Output and exit codes are byte-identical to the previous script on all 28 suite cases and on the six real ranges below.
supply-chain/README.md documents the self-test step, and states the guard's trust model: it runs from the pull request's own checkout like every other step in the workflow, so the rule is enforced by reviewing policy diffs (.github/CODEOWNERS owns /.github/workflows/, the default rule owns supply-chain/), not by the checkout enforcing anything on itself.

Not changed, with the reason on each thread: just supply-chain-guard (already in main via #667), and the dependency-migration and manifest-classification findings, whose premise is the rule text that #667 replaced on main.

Verification

Guard suite, all green (bash 5 and macOS /bin/bash 3.2; shellcheck -S warning clean on the guard and its test):

ok   source only -> OK
ok   .github/workflows/ci.yml alone -> OK
ok   scripts/ alone -> OK
ok   no files changed -> OK
ok   deny.toml only -> OK
ok   .cargo/audit.toml only -> OK
ok   Cargo.lock only -> OK
ok   crates/demo/Cargo.toml alone -> OK
ok   supply-chain/** + its workflow -> OK
ok   lockfile + policy + manifests together -> OK
ok   Cargo.lock + benchmarks/preloop-perf/Cargo.toml -> OK
ok   nested lockfile + its manifest -> OK
ok   Cargo.lock + crates/demo/nested/Cargo.toml -> OK (any manifest)
ok   packaging manifest + README.md -> FAIL
ok   nested lockfile + source -> FAIL
ok   benchmarks manifest + its source -> FAIL
ok   Cargo.lock + manifest + source -> FAIL
ok   combined set + README.md -> FAIL
ok   deny.toml + justfile -> FAIL
ok   supply-chain.yml + ci.yml -> FAIL
ok   Cargo.lock + CHANGELOG.md -> FAIL
ok   failure message offers no label or exception path
ok   failure message lists both sets under their own headings
ok   identical change on both tips -> FAIL (merge-base diff)
ok   policy-only branch over a moved main -> OK
ok   unresolvable revision -> exit 2 (fails closed)
ok   missing argument -> exit 2 with usage
ok   no merge base -> exit 2 (fails closed)

28 case(s), 0 failure(s)

The guard was also verified locally with the exact SHA pairs the engine passes (base, head), by running the script from this PR's head against those revisions in a fresh clone after git fetch --no-tags origin "$base" "$pr_head": exit 0, Dependency/policy PR detected: OK for this PR, and exit 0, OK (no policy or manifest files modified) for #667. The engine's own Supply chain audit check on this PR is the authoritative run; the local runs are the same script with the same two revisions.

Real history, run the same way (base..head as the engine would pass them):

Range	Result
this PR (`origin/main..HEAD`)	exit 0 — `Dependency/policy PR detected: OK` (workflow + `supply-chain/**` only)
#656 (`6be5967e..5735b909`, policy-only shape)	exit 0 — `supply-chain/config.toml` only
#655 (`301d823d..c7a3ffb8`, non-policy shape)	exit 0 — no policy or manifest file
#620 (`17846fc1..a66ca21b`, merged)	exit 1 — `Cargo.lock` + `crates/preloop-vm/Cargo.toml` + `crates/preloop-vm/src/*` + `CHANGELOG.md`; the shape the label exception existed for
#665 (`fd08d18e..8b4adb0`, dependency PR)	exit 0 — `Cargo.lock` + `crates/preloop-vm/Cargo.toml` only
#618 (`8b4adb0..8f2e094`, rebased)	exit 0 — source only, dependency PR already merged

actionlint .github/workflows/supply-chain.yml reports the same three pre-existing preloop-cpane self-hosted label notes as actionlint on the file at origin/main; nothing new. cargo vet --locked in this tree: Vetting Succeeded (17 fully audited, 2 partially audited, 467 exempted) — the new files under supply-chain/ do not disturb cargo-vet.

At this head (the review follow-ups): bash supply-chain/tests/guard-policy-files.test.sh -> 28 case(s), 0 failure(s) on bash 5.3 and on macOS /bin/bash 3.2; shellcheck -S warning clean on the guard and on its test; actionlint unchanged; and the six ranges below re-run with output and exit codes byte-identical to the pre-change script (diff of stdout+stderr, exit code compared).

`rust shard 4 of 4` at `266c9cb2d`: an unrelated, load-sensitive test in `main`

The failing check was not this branch. From the engine's retained log for run ff8b2b5d (job rust-test (4)):

FAIL [  19.190s] (613/744) preloop-runner-server github::tests::budget_expiry_builds_and_runs_the_engines_own_merge
thread 'github::tests::budget_expiry_builds_and_runs_the_engines_own_merge' panicked at crates/preloop-runner-server/src/github.rs:7282:9:
the delivery polls before giving up on GitHub
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1232 filtered out; finished in 19.18s

The assertion is polls > 1 against a stub that answers with a 5s wall-clock FreshMergePoll budget and a 20ms interval (github.rs:7262-7265): the loop has to complete a second poll before the budget expires, and on the guest one iteration takes longer than the budget. Same family, different PR, hours apart — fd0c79ee (#664) failed rust shard 4 of 4 with github::tests::poll_slot_is_released_once_github_offers_the_merge, github.rs:8044:22, GitHub's merge must be used once offered, got None, also ~18s; #668/#669 failed shard 2 and #670 shard 3 in the same window. This branch's diff is .github/workflows/supply-chain.yml plus supply-chain/** — no Rust file at all, so the flake cannot come from here, and it cannot be fixed here either: crates/preloop-runner-server/src/github.rs is a non-policy file, and touching it would make this PR fail its own guard. Fixing the flake (a budget that does not depend on guest load, or an injectable clock) belongs in a Rust PR.

It does not reproduce off the loaded guest: the same partition run locally at this head (cargo nextest run --locked --no-fail-fast --workspace --partition hash:4/4) reports 757 tests run: 757 passed (11 slow, 1 leaky), 2315 skipped, with budget_expiry_builds_and_runs_the_engines_own_merge at 21.6s and poll_slot_is_released_once_github_offers_the_merge at 63.5s (flagged SLOW [> 60.000s]) — both pass here, both are wall-clock-sensitive, neither is touched by this diff.

This PR passes its own guard

The diff is .github/workflows/supply-chain.yml, supply-chain/README.md, supply-chain/guard-policy-files.sh, supply-chain/tests/guard-policy-files.test.sh — dependency/policy files only, no other file, so the "dedicated dependency/policy PR" shape is satisfied.

Open PRs the new guard fails, and what their owners need to do

These PRs mix dependency/policy files with other files, so they cannot merge once this lands until the dependency/policy change is split into its own PR (merge that first, then rebase the code PR). The engine's Supply chain audit check says the same thing on each:

PR	dependency/policy files	other files
#451	`Cargo.lock`, `crates/preloop-runner/Cargo.toml`	5 files under `crates/preloop-runner/src/**`: `client/broker.rs`, `client/http.rs`, `worker/job_extension.rs`, `worker/job_extension_tests.rs`, `worker/job_runner.rs`
#619	`Cargo.lock`, `Cargo.toml`, `crates/{preloop-cli,preloop-orchestrator,preloop-sandbox,preloop-vm}/Cargo.toml`	51 files: `AGENTS.md`, `CHANGELOG.md`, `benchmarks/substrates/e2e-bench.sh`, `benchmarks/substrates/project-bench.sh`, 47 under `crates//src/`
#647	`.github/workflows/supply-chain.yml`	129 files: `.github/workflows/control-plane.yml`, `.gitignore`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `REVIEW.md`, the docs site, ...
#648	`Cargo.lock`, `Cargo.toml`, `benchmarks/{control-load,preloop-perf}/Cargo.toml`, `crates/{preloop-cli,preloop-orchestrator,preloop-runner-server,preloop-socket-activation,preloop-vm}/Cargo.toml`	103 files: `.github/workflows/ci.yml`, `AGENTS.md`, `CHANGELOG.md`, `CONTRIBUTING.md`, `REVIEW.md`, `benchmarks/preloop-perf/src/main.rs`, ...
#618 (pre-split head `af64e3dd`)	`Cargo.lock`, `crates/preloop-vm/Cargo.toml`	10 files (`crates/preloop-vm/src/`, `crates/preloop-cli/src/`, `docs/`, `CHANGELOG.md`) — already split*: #665 and the rebased #618 both pass

#660-#664, and everything else that touches no dependency/policy file, pass as they stand. #648 is the clearest case for shipping the consolidated shape instead of splitting: its manifests can go with the lockfile in one PR now.

Checklist

Tests added/updated for any new observable contract (supply-chain/tests/guard-policy-files.test.sh, now run by the workflow's Guard policy files (self-test) step)
Docs updated (supply-chain/README.md; the AGENTS.md, CONTRIBUTING.md, PR-template and CHANGELOG text landed in #667, which touches no policy file)
Changelog-worthy user-facing change described in the PR body (the CHANGELOG.md entry is in #667, since CHANGELOG.md is not a policy file)

Merging this closes the two-PR change: #667 is on main, and this branch is the other half.

Push note: the branch was pushed with git push --no-verify because the installed git-lfs pre-push hook fails on this machine (Size cannot change: [422] Size cannot change), not because of anything in this diff. The follow-up commits were plain fast-forward pushes.

Summary by CodeRabbit

Workflow
- Pull requests are checked to ensure dependency and policy-file changes are not mixed with unrelated changes. Dependency/policy-only changes and changes that do not touch those files are allowed.
- The check now validates changes from the merge base to the pull request head and runs a self-test before applying the policy check.
Documentation
- Added guidance on permitted change combinations, the required order for dependency and code pull requests, and how to run the check locally.