fix(policies): evaluate redacted materials against the content that was stored

Merged
JI
2mo ago
+921-336

Secret redaction (#3368) only rewrites the copy of a CHAINLOOP_AI_CODING_SESSION that leaves the machine. Policies were still evaluated against the untouched original on disk, having access to any potential secret that could have leaked into the AI session.

The invariant now implemented is that the policy engine receives exactly the bytes that were stored.

Changes

  • Crafters that do not store the artifact verbatim implement a transformCrafter seam and report the bytes they stored. materials.Craft surfaces them as CraftResult.EvaluableContent, and the crafter passes them to both policy verifiers via the new policies.WithMaterialContent per-call option.
  • Content resolution fails closed: a material annotated chainloop.material.redacted=true returns ErrRedactedContentRequired rather than falling back to the file on disk. This is uniform across CAS backends, inline included.
  • chainloop policy devel eval threads the same content through, so a policy is developed against the input the real run produces.
  • The bundled testdata policy now detects [REDACTED:<rule>] placeholders and the redaction annotations, since the raw pattern it looked for is no longer present in its input.

Materials that were not transformed are unaffected and hold nothing extra in memory: their content still resolves from the inline copy or the file on disk. This also closes a gap for skipUpload materials, whose sanitized bytes were previously retained nowhere at all.

Notable consequences

A redacted material's policy input can no longer be reconstructed from persisted crafting state alone, so a future push-time or server-side material evaluation must plumb the bytes through as well. materials.Craft now returns *CraftResult and policies.Verifier.VerifyMaterial takes variadic options — both breaking for external importers of these packages.

Not covered here

Two gaps remain that are properties of redaction rather than this plumbing: --skip-secret-redaction stores the session as captured, and secrets counted as report.Unlocated (landing in a protected path, or spanning two JSON leaves) are not redacted in the stored copy either. Detecting placeholders in the upstream ai-coding-session-no-secrets policy is the second part of the issue and lives in another repository; that policy should keep its pattern matching alongside the placeholder check, since pattern matching remains the only thing that catches the secrets redaction could not reach.

AI disclosure

Produced with assistance from Claude Code; every commit carries an Assisted-by: Claude Code trailer.