feat(artifact-cas): verify uploaded content against the declared digest

Merged
JA
1mo ago
+949-507

The Artifact CAS keyed stored artifacts by their client-declared SHA256 without ever
checking that the uploaded bytes hash to that digest. Only the OCI backend verified, via
its own layer-digest check. On S3, S3 access point and Azure Blob any identity holding
artifact:upload could store arbitrary content under an arbitrary digest key.

The CAS now streams each upload to a local staging file, hashes it while receiving, and
hands the backend a file only once its digest matches the one the client declared. A
mismatch is rejected with InvalidArgument and nothing reaches the backend.

What changed

  • A single spill-then-upload path shared by all backends, replacing the per-backend
    streaming branch and the StreamingUploader interface. Verification is no longer
    something each backend implements, and any future backend inherits it.
  • Staging files are unlinked as soon as they are created and used through the open
    descriptor, so the kernel reclaims the space whenever the process exits, including when
    it is killed mid-upload. Nothing needs to sweep or reap the staging volume.
  • staging_dir is required: the CAS refuses to start without one and proves it is
    writable before serving traffic, so a misconfiguration fails at boot rather than
    producing a healthy service that rejects every upload.
  • Helm: a dedicated staging emptyDir mounted unconditionally at a fixed path. It sets no
    sizeLimit, since exceeding one evicts the Pod rather than failing the upload that
    overran it.
  • Staging on disk keeps the service's own memory use independent of artifact size. The
    OCI backend still requires the whole layer in memory, unchanged here.
  • The S3 backend relies on this service-layer verification rather than a per-object
    checksum precondition, which cannot be expressed for multipart uploads.

Addresses PFM-6988.

This contribution was produced with the assistance of Claude Code.