feat: show the encryption state of artifacts and add routes to encrypt the files stored before
For humans
Customers cannot see which artifacts are stored unencrypted, and cannot encrypt the old files.
Adds an encryption state per artifact with page notices, and routes that encrypt the old files.
Check: the routes, the checksum rules and the memory limit. Stacked on #10746.
Why
Customers who enable storage encryption encrypt only files stored from then on. Without a status they cannot see which artifacts are still stored without encryption. After a key change they cannot see which encrypted artifacts the current key can no longer read. Source: INFP-740.
Files stored before encryption was enabled also stay in plaintext, and regenerating an artifact does not encrypt them. An operator has no way to encrypt them. The routes of this pull request let an administrator do it through the API server, which holds the key, with a client that sets the pace. The client is the infrahubctl storage encrypt command: opsmill/infrahub-sdk-python#1535.
Base branch: sas-storage-encryption (PR #10746, not merged). Retarget to develop after it merges.
Non-goals: a list or filter of unreadable artifacts, key rotation, checking the content of a file.
What changed
- Stored value: new read-only attribute
encryptiononCoreArtifact:plaintext(default),encryptedorunreadable. The check treats an artifact with no value asplaintext. - Check: new
POST /api/artifact/{artifact_id}/encryption. It reads the first bytes of the stored file. They name the key that sealed it, so the check tells a file sealed with another key, a file with no key available, an unsupported format and a truncated file. It does not decrypt, so it does not find modified content. Reasons:no_key,unknown_key,unsupported_format,truncated,missing. - Writing the value: when the result differs from the stored value, the endpoint stores it if the caller may update artifacts and the branch accepts writes. Otherwise it returns the result. A missing file leaves the stored value.
- Worker call: the SDK skips read-only attributes when it saves a node, so the task worker cannot write the value. After it saves an artifact, the worker calls the endpoint, and the API server is the only writer. A failed call is logged and leaves the stored value.
- Flag for the UI:
/api/infogainsstorage_encryption_enabled. It follows the anonymous access setting, unlike/api/config, which is always public. - Artifact page: it sends the check when it opens, except to an anonymous visitor and while a past date is viewed.
- Warning banner: the artifact is
plaintextand encryption is enabled. - Green
encryptedbadge with a lock, next to the status badge: a check confirmed the artifact isencrypted. A storedencryptedvalue alone shows nothing, because it can be out of date. - Error banner: the artifact is
unreadable, with the reason when the check gave it. When the file is missing, anunreadableartifact says so. - A missing file gives no other indicator.
- Past date: no check is sent and a check result of the current file is ignored. The page shows the warning or the error for the value stored at that date, without the reason, and no badge.
- Warning banner: the artifact is
- Anonymous visitors: the check needs authentication, so they see the stored value, never the badge and never the reason.
- Encrypting files stored before encryption was enabled:
GET /api/storage/objectslists the stored objects in pages. The cursor follows the last name scanned, so a file that disappears during a scan does not end the listing early.POST /api/storage/objects/encryptencrypts up to 500 objects, andPOST /api/storage/object/{identifier}/encryptone object. The three routes need the super admin permission and answer 409 when encryption is not enabled or the key is missing.- For each object the server reads the header, finds the recorded checksum, reads the file, checks the checksum, seals the file, writes it back under the same identifier (local disk: temporary file,
fsync, rename; S3: put) and confirms the header. The outcomes areencrypted,already_encrypted,skipped_unverified(no node records a checksum, for example an orphan file),skipped_checksum_mismatch,skipped_too_large,missingandfailed. - A byte budget per API server process,
INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MEMORY_BUDGET(MB, default 600), bounds the memory used while files are encrypted. A request that cannot get memory is answered 429 withRetry-After.INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MAX_SIZE(MB, default 100) is the largest file the migration reads. - The first page of a listing removes temporary files of this feature older than one hour, which a killed server may leave in local storage.
--dry-runof the client therefore does not leave local storage untouched. - After the files of a request are encrypted, the server stores
encryptedon the default-branch artifacts whose current file is one of them (outcomesencryptedandalready_encrypted): one query for the artifacts of a request and one write for each artifact whose value differs. A failed write is logged and does not change the outcome of the file; a new pass over the same files writes the value again. An artifact that exists only on another branch keeps its value until its check runs.
- Unchanged: no change to upload, download or encryption itself, no database migration, and no change to the SDK submodule.
Screenshots
Artifact page, local stack, signed in as an admin. The encrypted state is on commit 2e68819, the others on 815e64b. The warning and error banners did not change between the two.
| State | Screenshot |
|---|---|
| Encryption enabled, artifact stored before it was enabled | |
| Encryption enabled, a check confirmed the artifact is encrypted: badge in the header | |
| Encryption enabled, the key changed after the artifact was stored | |
| Encryption disabled, artifact stored without encryption: no notice |
How to review
- Start with
backend/infrahub/storage_encryption/status.pyand, for the routes,backend/infrahub/api/storage/encryption_migration.py,backend/infrahub/storage_encryption/migrate.py,queries.py,artifact_values.pyandmemory_budget.py; thenbackend/infrahub/api/artifact.py(check_artifact_encryption), thenbackend/infrahub/artifacts/encryption.py, thenfrontend/app/src/entities/artifacts/domain/rules/get-artifact-encryption-notice.ts. - Decisions for reviewers:
- A check from a feature branch writes the value on that branch, because
CoreArtifactis branch-local. Not confirmed: that this shows as a change in a proposed change. - The worker uses the SDK private methods
_postand_request_headers. No other code inbackend/infrahubdoes. - Any signed-in user can check any artifact, as with the content endpoint. There is no view permission check.
POST /api/artifact/generate/encryptionis routed to the check, not to the generation of a definition namedencryption, because the check route is registered first.- The check adds one request per artifact generation and one per artifact page view.
- The recorded checksum of a stored file is taken from any node that points at the file, on any branch and in any state. A deleted node still counts. A file is encrypted only if its content matches one recorded checksum.
- The routes replace files in place and never delete a file other than the stale temporary files.
- A check from a feature branch writes the value on that branch, because
How to test
uv run ruff check . --exclude python_sdk && uv run ruff format --check --exclude python_sdk .
uv run pytest backend/tests/unit -n 4 # 5291 passed, before the review fixes
uv run pytest backend/tests/unit/storage backend/tests/unit/storage_encryption # 90 passed, after the fixes
uv run pytest backend/tests/component/api backend/tests/component/storage -n 3 # 211 passed, 1 expected failure, 1 fixed (below)
uv run pytest backend/tests/component/proposed_change -n 3 # 72 passed
uv run pytest backend/tests/component/api/test_61_artifact_encryption.py backend/tests/component/api/test_62_artifact_encryption_permissions.py backend/tests/component/api/test_11_artifact.py -n 3 # 25 passed
uv run invoke docs.validate
cd frontend && pnpm exec biome ci .
cd frontend/app && pnpm knip && pnpm exec betterer ci
cd frontend/app && pnpm exec vitest run src/entities/artifacts # 31 passed
- Existing tests changed:
test_info_endpointpinned the exact fields of/api/info. Two frontend tests built an/api/inforesponse by hand. A git test fixture now mocks the new worker request. - Loaded runs: a git and import run together with the full frontend suite hit its 30-minute limit. 5 import tests failed, and all passed when run alone. A different single test failed in one rerun and passed on the next. 49 frontend tests in 3 files failed under load, and 61 tests in those files pass alone. Not root-caused.
- Teardown error:
test_artifact_generate_python_newends with an unmatchedGET /api/admin/version. It also happened on the first test of a run before this change. Not root-caused. - Guards: each new test fails when its fix is removed. Without the update permission check, the permission test fails. Without the key comparison or the off-mode rule, 3 unit tests fail. Treating
AccessDeniedas missing fails the S3 test. Without the missing-file branch, 2 frontend tests fail. - Live stack (Docker Compose, server and two task workers, local storage): checked in Chrome in four states.
- Encryption on, plaintext artifact: warning.
- Encryption on, encrypted artifact: green notice (after a signed-in check).
- A different key: error.
- Encryption off, plaintext artifact: no notice.
- A worker generated an artifact with encryption on, and the server stored
encryptedon it. - Anonymous visitors saw a green notice for an unreadable artifact before the notice required a live check. That is fixed.
- Past date: checked in Chrome on the code of
a0e74f8, before its commit (warning shown from the stored value, no check request). The badge was checked in Chrome on the code ofb3ca3b8, before its commit.
Tests of the routes
- Unit tests of
backend/tests/unit/storageandbackend/tests/unit/storage_encryption: 373 passed. Component teststest_60totest_65, single process: 49 passed. The whole backend unit suite with-n 4: 5,581 passed. - Each new test fails when the behaviour it covers is broken (a deliberate breakage of each rule in a scratch copy; the mutation tables are in local reports). For the listing cursor, 5 tests fail with the old rule.
- Tests check that the single route and the batch route give the same outcome for every refusal (
test_migrate_parity.py,test_64). - Stack with real task workers, standalone client against the routes (not the SDK command), 157,000 files (50,000 artifacts with 3 versions, 5,000 file objects, 2,000 orphan files, 28.9 GB), S3 (Garage) on a clean VM with 5 foreground requests per second: 155,000 files encrypted in 601 s, 2,000 orphan files unchanged, 0 of 50,300 downloads failed, the database peaked at 3.6 GiB. The first one-request-per-file prototype did 26.5 objects per second under the same conditions.
- Memory: a batch of 50 files of 90 MiB grew the server by 246 MiB, and four such batches at once by 692 MiB (before the byte budget: 459 and 1,202 MiB).
- The stored value after a migration: 8 component tests and 1 expected failure (
test_66). With the artifact step switched off, 5 of them fail withplaintextwhereencryptedis expected. On a local stack with 1,000 artifacts (3,150 files), the SDK command left all 1,000 valuesplaintextbefore this commit andencryptedafter it, with every file encrypted and 0 failed downloads. The migration took 4.5 s before and 8.5 s after (3 runs each, about 4 ms and 3 database statements per updated artifact). Not measured: S3 and 50,000 artifacts. - Kill tests on the small dataset (6,300 files): killing the client with SIGKILL and killing the server during a run lost nothing and encrypted nothing twice. No half-written file appeared under a real name in three kills during large batches.
Impact & rollout
- Backward compatibility: none broken. One new optional attribute with a default, four new endpoints, one new field in
/api/info, two new settings. - Performance: the check reads at most 41 bytes of a file. It adds one request per artifact generation and one per artifact page view. The migration loads the API server and the database while it runs: on the clean VM, with the SDK command, the database used 152% CPU on average (peak 637%) and the foreground 95th percentile was 1.2 times its baseline. Writing the stored value adds about 4 ms for each artifact whose value changes. Speed on a remote S3 service is not measured.
- Config/env changes: two new settings with defaults, declared in both compose files:
INFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MEMORY_BUDGET(600) andINFRAHUB_STORAGE_ENCRYPTION_MIGRATION_MAX_SIZE(100). With several API server processes the worst-case memory is the budget times the number of processes. - Deployment notes: the SDK protocol file for
CoreArtifactlives in thepython_sdksubmodule. The change is in opsmill/infrahub-sdk-python#1526, which is not merged. The submodule pointer is bumped to its commit (4fbf74e1, which builds on the previous pointer), and must be bumped again to the merge commit once #1526 is merged.
Not done or not verified
- Not verified: that existing artifacts get the default value when a database is upgraded. The check and the page tolerate a missing value.
- Not verified: that a write on a feature branch shows in a proposed change.
- Not tested: a failed worker call is only logged, and the S3 header read against real S3 or MinIO.
- Not run: the rest of the git component suite, which the 30-minute limit cut short. The artifact tests of that suite passed.
- No view permission check on the new endpoint, and no list or filter of unreadable artifacts.
- An artifact that exists only on a branch other than the default keeps its stored
encryptionvalue after the migration, until its check runs (the artifact page, or the worker when the artifact is generated). Supporting it needs a branch lookup, per-branch permission and status checks, and more writes. - The commit that stores the value (
426e49e68c) and the regenerated schema were not reviewed by thecode-revieworsecurity-reviewskills. The reviews below cover60ac20562f(code review) and45b41d53e6(security review). - Not measured: a remote S3 service (every S3 run used Garage on the same machine); several API server processes; the kill of the server on S3 and at the scale of 157,000 files.
- One foreground request failed with "Server disconnected without sending a response" in each of two clean VM runs. The cause is not known.
- No integration test of the SDK command against these routes in CI. The command is in a separate pull request.
- The submodule pointer references the commit of the open SDK pull request #1526. After that pull request merges, the pointer needs a second bump to its merge commit.
The branch was rebased on the rebased #10746 and on develop (ee2d2dc402) on 2026-11-17. The commit hashes in the review sections below are those before the rebase. The content did not change: the only conflict was one import line in backend/infrahub/storage.py.
Code review
The code-review skill ran at level high, in eight rounds, each by an agent other than the author. In round 2 the skill reviewed the whole branch, so the agent kept only the findings that touch the fix or that it could verify. Round 4 reviewed a version that was not yet committed. Rounds 6 and 7 added mutation tests on a copy of the code.
| Round | Commit | Critical | Important | Minor |
|---|---|---|---|---|
| 1 | b869872 | 0 | 2 | 6 |
| 2 | 48dd723 | 0 | 3 | 7 |
| 3 | 815e64b | 0 | 0 | 5 |
| 4 | 4781b77, before its commit | 2 | 2 | 3 |
| 5 | 4781b77 | 0 | 2 | 5 |
| 6 | 815e64b..b3ca3b8 | 1 | 0 | 5 |
| 7 | 2e68819 | 0 | 0 | 3 |
| 8 | 71d011b | 0 | 0 | 3 |
fde5c63 removes one query option that the query client already sets. It had no review of its own.
Important findings:
- The green notice showed for an artifact whose file is missing: fixed in
48dd723. - A check from a feature branch writes the value on that branch: kept. It is the design. A change on a branch happens only when the stored value there is wrong, and generation on a branch writes the same attribute. Not confirmed: the diff effect. Listed above for reviewers.
- A plaintext artifact whose file is missing showed the warning: fixed in
815e64b. - The S3 error rule had no test: fixed in
815e64b. - Round 4, critical: a check result from the current view stayed displayed at a past date, and the hook test mocked
useQueryand proved nothing. Both fixed in4781b77. - Round 4, important: the notice at a past date described the current key, and no test covered it. Fixed in
a0e74f8, which shows the stored value at a past date. - Round 5, important: the documentation still said that the page checks each time it opens. Fixed in
a0e74f8. - Round 5, important: at a past date the page also hid the stored warning. Changed in
a0e74f8after a decision: it shows the stored warning or error. - Round 6, critical: a notice test passed without the past-date rule, because the notice never shows the encrypted state. Removed in
2e68819. The badge test covers the rule.
Minor, fixed: every S3 client error counted as a missing file, and the endpoint read the artifact at the at time of the request while it writes at the current time.
Minor, left as they are: the at parameter is accepted and ignored; an S3 store that hides missing keys from callers without list permission gives an error instead of missing; the worker uses two private SDK methods; the worker only logs a failed call; the page sends a check even when encryption is off; two functions repeat the same permission logic; the encryption detection rule is in two places; the notice component and the rule both handle a missing file; the S3 tests use hand-written fakes; no component test for an unreadable artifact whose file is missing; the endpoint returns an error when S3 fails for another reason.
Minor, left as they are (rounds 4 to 7): the query key has no date, so after a past date is cleared the earlier check result shows until the new check answers; the badge has the same green as the ready badge and only a tooltip explains it; the past-date condition is in the query hook and in the notice hook; the documentation does not say that a stored encrypted value shows nothing at a past date, and the warning at a past date follows the setting now; no notice-level test for a stored encrypted value at a past date; the removal of the encrypted field of UploadResponse from the schema is outside the UI change, and the backend model never had it. Round 8: a check result stays displayed in a tab after the user signs out in another tab, until a reload; no test for an anonymous visitor who signs in; the hook test replaces the whole authentication module.
Code review of the routes
The code-review skill ran once at level high, by an agent other than the author, on commit 60ac20562f, which is the first version of the routes. The fixes were amended into 45b41d53e6 and were not reviewed again. The skill reviewed the right checkout.
0 critical, 2 important, 9 minor.
Important, both fixed:
- On local storage a file that disappeared between the directory scan and
stat()made the listing return a short page with no cursor, so the client stopped and every later file stayed plaintext while the run looked finished. The cursor now follows the last name scanned (local and S3). - No test covered the same-branch and time-overlap rules of the recorded checksum query (one mutation left all 16 component tests passing). Six component tests on several branches now cover them.
Minor, fixed: the --dry-run text, raw exception text in reason, a note that the two checksum queries must stay in step.
Minor, left as they are:
- A cancelled batch runs to the end and a retry repeats it (the work is idempotent).
- A thread of the default executor waits up to 60 s per file for memory.
- The memory budget has no fairness: a large file can wait behind small ones until it gets a 429, and the client retries.
- On S3 a
putcan recreate an object deleted between the read and the write. - Every local listing page scans the whole directory, so a full pass costs about N squared over the page size (the docstring says so).
- One identifier containing
/or..makes the whole batch answer 422 (shared bucket with prefixes only).
Security review
The security-review skill ran from a clone whose origin/HEAD was set to sas-storage-encryption, so its diff is this change. It ran in seven rounds, each by an agent other than the author. In every round the agent did not start the skill's sub-tasks and read the diff by hand. In rounds 3 to 7 the skill reviewed the worktree and not the clone, so the agent discarded its input and applied its method by hand to the diff of the commits.
| Round | Commit | High | Medium | Low |
|---|---|---|---|---|
| 1 | b869872, 48dd723 | 0 | 0 | 3 |
| 2 | 815e64b | 0 | 0 | 2 |
| 3 | 4781b77 | 0 | 0 | 0 |
| 4 | a0e74f8 | 0 | 0 | 0 |
| 5 | 815e64b..b3ca3b8 | 0 | 0 | 0 |
| 6 | 2e68819 | 0 | 0 | 0 |
| 7 | 71d011b | 0 | 0 | 0 |
Low, left as they are:
- Route order: a definition named
encryptioncannot be generated through the API (listed above). storage_idis writable, so a user who may update artifacts can learn from the check whether a server path exists. The content endpoint already reads that path.- Any signed-in user can check any artifact, as with the content endpoint.
- The S3 test stand-in builds its errors by hand, so a real S3 backend that reports a missing object with another code would give an error. This fails safe.
- A missing file gives no notice for an artifact stored as
encryptedorplaintext. This is less visibility, not less protection.
Dropped after checks: forged requests from another site, setting the value from a client, injection in the worker URL, a mismatch of the stored value by someone who can write to the storage, and XSS in the notice text.
Security review of the routes
The security-review skill ran once, by an agent other than the author, on 45b41d53e6. Its diff was this commit; the agent applied its method by hand.
0 high, 0 medium, 1 low (7 findings dropped as unverified or speculative).
Low, left as it is: the checksum check proves that the content equals a recorded checksum, and the lookup accepts a checksum from any node on any branch. Someone who can already write to the storage and can create an artifact node could record the checksum of a changed file, and a later migration by a super admin would seal the changed content. The cost is that the checksum does not protect against tampering by someone with both rights. The migration never lets a file swapped during its own read-and-write window be sealed, and never rewrites a file that is already encrypted. A possible mitigation is to take checksums only from the default branch.
Checklist
- Tests added/updated
- Changelog entry added (
uv run towncrier create ...) - External docs updated (if user-facing or ops-facing change)
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content