feat: encrypt stored artifacts and files at rest
Artifacts and files are stored in plaintext, and edits in the storage go unseen.
Adds opt-in AES-256-GCM encryption at rest, off by default, for 1.13.
Review: dedicated key or login secret; no rotation; only new files are encrypted.
Why
Someone with access to the object storage (local disk or S3) can read every artifact and file object, and can change one without Infrahub noticing. Teams that must protect stored configuration need to switch encryption on without a migration.
Goal: an opt-in setting that encrypts new artifacts and files and refuses a modified one, with a key that can be separate from the login secret. Source: INFP-740. Decision to target develop and test it there: Slack thread.
Non-goals: key rotation, encrypting files stored before, a size limit on uploads, a migration or CLI.
What changed
- Setting:
INFRAHUB_STORAGE_ENCRYPTION_ENABLED, defaultfalse, set on the API servers. - Key:
INFRAHUB_STORAGE_ENCRYPTION_KEYis new and optional. When set, the key is derived from it andINFRAHUB_SECURITY_SECRET_KEYplays no part. When unset or empty, the key is derived fromINFRAHUB_SECURITY_SECRET_KEY. With encryption on, startup refuses a dedicated key under 32 characters, and does not fall back to the secret. Without a dedicated key it refuses a generated, published or short secret (this replaces the earlier startup rule). - New files: sealed with AES-256-GCM. The key is derived with HKDF from the key that applies and never stored. The header and the storage id are authenticated, so a modified or moved file returns 409.
- Files stored before: stay plaintext. They are served only if they match the checksum recorded for them, even when
INFRAHUB_STORAGE_VERIFY_ARTIFACT_CHECKSUMisfalse. A file with no recorded checksum is served as it is. - Turning it off: only stops encrypting new files. Files encrypted earlier stay readable while the key is unchanged. Without a usable key they return a 500 error that names the setting.
- Regeneration: an artifact is encrypted when its content changes. Unchanged content keeps its stored file, unless a check of the stored file is requested and the file is missing or refused. An old plaintext file is not deleted.
- Unchanged: no database change, migration, API or CLI. Task workers need no setting: they go through the API.
- One reader:
developalready had a checksum reader (ArtifactContentReader). The three download endpoints now share it, so there is one query and one pair of errors.
How to review
- Start with
backend/infrahub/storage_encryption/crypto.py, thenbackend/infrahub/storage.py(read), thenbackend/infrahub/artifacts/content.py. - Check the off path: with the setting off and no encrypted file, behaviour must match
develop. Only one extra check differs: whether a file is a complete encrypted envelope. - Decisions for reviewers:
- Without a dedicated key, the key comes from the secret that signs login tokens. A guessable key of 32 characters or more can be guessed offline from a stored file, and this holds for the dedicated key too. The docs warn and give
openssl rand -hex 32. - Setting a dedicated key after files were encrypted with the secret, or removing it, is a key change: the earlier encrypted files are refused with 409. Nothing reads with both keys. Decide whether that is acceptable for 1.13.
- A 50 MB upload holds about 100 MB at peak with encryption on (under 1 MB off). The endpoint had no size limit before; the docs list this.
- Old plaintext files stay after regeneration. There is no tool to encrypt them in place.
- Without a dedicated key, the key comes from the secret that signs login tokens. A guessable key of 32 characters or more can be guessed offline from a stored file, and this holds for the dedicated key too. The docs warn and give
How to test
uv run ruff check . --exclude python_sdk && uv run ruff format --check --exclude python_sdk .
uv run pytest backend/tests/unit -n 4 # 5277 passed
uv run pytest backend/tests/component/api backend/tests/component/storage -n 3 # 206 passed, 1 expected failure
uv run invoke docs.validate
- Earlier commits only: the private repo run (
communitypinned to an earlier commit: 116 storage tests, 15 Enterprise component tests, 136 Enterprise unit tests; one UDP log-forwarding test fails on macOS and the TCP log-forwarding module hangs, as on the unchanged pinned commit) and the other component and integration suites (artifact, git, diff, file object: 327 passed, 1 skipped, 2 expected failures; 16 integration tests passed). Not re-run on the dedicated key commit. - Live stack, dedicated key (Docker Compose, image built from this commit, server and two task workers, local storage):
- Encryption on with the published example secret and no dedicated key: the server exits with the new message.
- Dedicated key set, secret still the published example: the server starts. The stored file header holds the fingerprint of the dedicated key, and the API serves the content decrypted.
- Dedicated key unset, random secret: the header holds the fingerprint derived from the secret. The file encrypted with the dedicated key is refused with 409.
- Dedicated key of 7 characters: the server exits with
INFRAHUB_STORAGE_ENCRYPTION_KEY must be at least 32 characters long. - Dedicated key set again, secret unchanged: the file encrypted with it is served, and the file encrypted with the secret is refused with 409.
- A task worker generated one artifact: its stored file carries the dedicated key fingerprint, holds no plaintext, and the served content matches the recorded checksum.
- Live stack (Docker, real task workers, earlier commit): 20 artifacts generated with encryption off, then on. The 18 older ones stayed plain and were served after a checksum check. The 2 regenerated ones were encrypted. 15 attack scenarios (flip, truncate, append, swap, replace with plaintext, delete, wrong secret) returned 409 or 404 as expected. S3 (local emulator): 10 of 10.
- Guards: each new test fails when its fix is removed (checked by reverting each fix): binary download, forced checksum, key for reading, marker handling, decrypt while off, upper-case secret. For the dedicated key: ignoring it when deriving the key fails 4 unit tests, and skipping its length check fails 3.
- Performance (one process, plain and encrypted interleaved): encrypting adds about 0.3 to 0.5 ms per MB on write and 0.3 ms per MB on read. A comparison against
developover HTTP with encryption off could not resolve a difference: the same code varied by about 28% between runs.
Impact & rollout
- Backward compatibility: none broken. Off by default. Existing files are served unchanged.
- Performance: see above. The cost that matters is memory: about twice the file size at peak while encrypting or decrypting.
- Config/env changes: new
INFRAHUB_STORAGE_ENCRYPTION_ENABLEDandINFRAHUB_STORAGE_ENCRYPTION_KEY, both declared in both compose files. Before turning encryption on, setINFRAHUB_STORAGE_ENCRYPTION_KEY, orINFRAHUB_SECURITY_SECRET_KEYwithout it, to 32 or more random characters, the same on every API server. - Deployment notes: back up the database and the storage together. A restore needs the key that was in use.
Not done or not verified
- No browser check of the UI. No backup and restore test with encryption.
- The live stack test of the earlier commits (20 artifacts, attack scenarios, S3) was not re-run on the dedicated key commit.
- Only the API and storage component suites were run on the dedicated key commit, not the full community suites.
- A task worker without the dedicated key was not tested: both compose files give the server and the workers the same variables.
- No key rotation, no reader for both keys, and no upload size limit.
The branch was rebased on develop (ee2d2dc402) on 2026-11-17. The commit hashes in the review sections below are those before the rebase. The content did not change: the only conflict was one import line in backend/infrahub/storage.py.
Code review
The code-review skill ran at level high, in five rounds, each by an agent other than the author, each on the commits up to that point. In round 2 the skill's first run reviewed another checkout (wrong working directory); the agent discarded it and reviewed the verified diff. In round 5 the agent reviewed only the new commit (d7a8c51). The skill's output did not name the repository or diff, so the agent checked the checkout and verified each of the 7 skill findings against the code: 2 were dropped, 5 are minor.
| Round | Commit | Critical | Important | Minor |
|---|---|---|---|---|
| 1 | first version rebased on develop | 0 | 6 | 11 |
| 2 | 95db617 | 1 | 2 | 6 |
| 3 | 68b4193 | 0 | 0 | 11 |
| 4 | bd2491c | 0 | 0 | 8 |
| 5 (dedicated key) | d7a8c51 | 0 | 0 | 5 |
Critical and important findings:
- Decryption failures were not logged: fixed. The log has the storage id, the reason and the key fingerprint.
- The opt-out setting let a plaintext replacement through: fixed. The check is forced on while encryption is on.
- A file encrypted earlier was served as ciphertext after turning encryption off: fixed. It decrypts.
- No test for encrypted file objects: fixed.
- Upload memory: kept. No limit existed before; measured and documented.
- No key rotation: kept. Documented limitation; not in scope.
- Critical (round 2): with encryption off, a key from the published secret let anyone with storage write access forge an artifact. Fixed in
68b4193. - Docs overstated protection after turning off, and startup checked the secret late: fixed.
- Round 5: none.
Minor, left as they are: a test computes its expected value with the code under test; quote-stripping order in the secret check; one docs sentence is broader than the code; the object storage is built twice at startup; seal uses private fields of the key class; the key length has no constant; the example secret is copied in other files; the guard condition repeats in the reader. Round 5: a blank dedicated key is read as unset, so encryption falls back to the secret without a warning; two functions each test whether the dedicated key is set; the dedicated key is checked for length only (the published example value and a key padded with spaces are accepted); no test reads the key from the environment variable; the published example value and the test key are defined again in the tests.
Security review
The security-review skill ran from a clone whose origin/HEAD is develop, in five rounds, each by an agent other than the author. In round 1 the skill's context held a larger diff, so the agent reviewed the verified 29-file diff. In round 2 the skill did not start (working directory was not a repository) and the agent reviewed the verified diff by hand. Rounds 3 and 4 ran the skill on the verified diff. Round 5 ran the skill on the whole-PR diff (29 files); the agent scoped its findings to the new commit, read that diff and ran a script instead of starting sub-tasks, and found nothing in older code.
| Round | Commit | High | Medium | Low |
|---|---|---|---|---|
| 1 | 70e8d74 | 0 | 1 | 4 |
| 2 | 95db617 | 0 | 1 | 5 |
| 3 | 68b4193 | 0 | 0 | 3 |
| 4 | bd2491c | 0 | 0 | 2 |
| 5 (dedicated key) | d7a8c51 | 0 | 0 | 3 |
Medium findings:
- The key derives from the login secret (or the dedicated key) with a fast hash, so a guessable value of 32 or more characters can be guessed offline. Kept: the design stays; the docs warn.
- Round 2: with encryption off, a key from the published example secret let anyone with storage write access forge an artifact. Fixed in
68b4193: no key comes from a published, short or generated secret, and decrypted content is still checksum-checked while encryption is off. - Round 5: none.
Low, left as they are: an object with no recorded checksum can be replaced with plain content (documented); an older encrypted copy can be put back (documented); the key fingerprint is in headers and logs; a plain file that starts like an encrypted one is refused (documented); the 500 error names the key settings; the published-secret check covers the shipped forms only; a file object with a checksum of another length is served unchecked. Round 5: the dedicated key is checked for length only, so the published example value is accepted as a dedicated key; the length counts whitespace, and a blank key silently falls back to the secret (documented); the key fingerprint in each file lets someone with read access test guesses of a weak dedicated key offline (the ciphertext allows the same, and the docs warn).
Dependency scan
- Adds: direct requirement
cryptography>=50,<51inpyproject.tomlanduv.lock. It was already locked at 50.0.0 as a transitive dependency, so no new package is locked. - Scan: Trivy 0.74.0 filesystem scan, vulnerability database 2026-11-06 12:55 UTC, branch
70e8d74b2cagainst base100a1fa87d. The manifests are identical in the commit that was scanned and in the currentdevelop. The dedicated key commit changes no manifest or lock file, so it was not scanned again. - New findings: 0 (41 in both). Critical and high: none. Medium, low and unknown: none.
Checklist
- Tests added/updated
- Changelog entry added (
uv run towncrier create ...) - External docs updated (if user-facing or ops-facing change)
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content