feat: add infrahubctl storage encrypt for files stored before encryption was enabled
For humans
Files stored before encryption was switched on stay unencrypted, and nothing fixes them.
This adds an infrahubctl command that encrypts them through the server, with progress and resume.
Check: it needs server routes that are not released, and the speed proof is partly noisy.
Why
Files stored before storage encryption was enabled stay in plaintext, and regenerating an artifact does not encrypt the old files. An operator has no way to encrypt them.
Goal: a command that encrypts the stored files through the API of an Infrahub server, which holds the key. The operator sees the total and the progress, and can stop and resume at any time, as with infrahub upgrade for database migrations.
Non-goals: no key on the client.
Source: INFP-740. The server routes are added in opsmill/infrahub#11561, which is not merged.
What changed
- New command
infrahubctl storage encrypt. It lists the stored objects, counts them first, and prints the total. - The progress line shows done out of total, percentage, objects per second, time remaining, bytes, the concurrency limit, 429 answers and retries.
- It sends 50 objects per request with several requests in flight (
--concurrency, default 8, at most 64). The number in flight goes down on a 429 answer and up on success. It waits on 429 answers usingRetry-After(one pause is capped at 300 s) and retries network errors and 502, 503 and 504. - A state file keeps the position, so a run stopped with Ctrl-C (exit 130), SIGTERM (exit 143) or a kill resumes where it stopped. A report file lists every object that was not encrypted.
- Other options:
--dry-run,--rate,--before,--limit,--allow-unverified,--no-count,--restart,--priority. - The position follows the cursor that the server returns for each page, so a page that is empty but has a cursor does not stop or repeat the run. A server that returns the same position twice stops the run with an error (exit 2).
- A password login that expires during a long run is renewed once and the request is sent again.
- The command builds its own
httpxclient from the SDK configuration, because the SDK opens a new connection for each request and retries 429 answers itself, which would hide them from the pacing.
How to review
- Start with
infrahub_sdk/ctl/storage/migration.py(state, pacing, retries), thenencrypt.py(command, signals) andprogress.py(progress line). - Decisions to confirm: the own
httpxclient built from private SDK members; unlimited waiting on 429 answers (Ctrl-C and SIGTERM stop it); state and report files in the current directory by default; the count pass before the run.
Testing
- 202 tests in three files;
tests/unit/ctl: 647 passed, 1 expected failure, on Python 3.14.7 and 3.10.21. - Each new test fails when the behaviour it covers is broken and passes with it: 62 deliberate breakages for the progress work, 71 for the first tests, and one for each fix after the code review and for the cursor change (7 of the 8 cursor tests fail with the old loop; the eighth guards the
--limitcut and fails with a naive fix). The reports are local files and are not published. - ruff, ty, mypy, rumdl and vale pass.
docs-generategives the same page on a second run. - Stop and resume on local disk, small dataset (6,300 files): SIGKILL, SIGTERM and Ctrl-C each followed by a rerun. The rerun printed
Already handled in earlier runs: 1,000andTo process: 5,300, ended at 6,300 of 6,300, and the audit passed (2,300 of 2,300 downloads match, no temporary file). - Speed on a laptop, S3 (Garage on the same machine), small dataset: the median of the command was 98.9% of the standalone benchmark client over 13 runs. The baseline alone ranged from 234 to 304 objects per second because other jobs loaded the laptop. Over the first five planned pairs it was 93.4%, and over the runs after the other jobs stopped 97.6%. Against a mock server (100,000 objects) it was 99.7%. This does not prove or rule out a regression.
- Count pass: 0.25 s for 6,300 objects on S3. For 157,000 objects: 2.8 and 2.9 s on local disk, 6.8 and 8.5 s on S3.
- Clean VM, 157,000 files in S3 (Garage), 5 foreground requests per second: the command took 579 s (267.6 objects per second) and the standalone client 601 s (258.1). The audits passed: 155,000 files encrypted, the 2,000 orphan files unchanged, 0 of 50,300 downloads failed. This ran the version before the count pass and the review fixes.
Impact & rollout
- Backward compatibility: a new command; no existing command changes.
- Config: none. The account needs the super admin permission, and the server needs the storage routes of opsmill/infrahub#11561, so this targets
infrahub-develop. - Dependencies: none added (
pyproject.tomlanduv.lockare unchanged).
Not done or not verified
- The clean VM run used the version before the count pass and the review fixes. The comparison of the final version on S3 was disturbed by other jobs.
- No remote S3 service: every S3 run used Garage on the same machine.
- No integration test against a real server in CI: the tests use mocks. The runs against a real server used a local stack built from the server branch.
- Python 3.11 and 3.12 were not run. Python 3.13 ran only on the code before the review fixes. Windows was not checked.
- The server stores
encryptedon the default-branch artifacts whose file it encrypts. An artifact that exists only on another branch keeps its stored value until its check runs.
Code review
The code-review skill ran once at level high, by an agent other than the author, on commit 4dcc5c72. The fixes, and the later change of the paging position (follow the cursor of the server, found while reviewing the server routes), are amended into the final commit 1b8c6d87 and were not reviewed again. The skill reviewed the right checkout. The agent verified its 10 findings (2 dropped) and added its own.
0 critical, 3 important, 12 minor.
Important, all fixed:
- An expired login token was never renewed, so a long run stopped on a 401.
- A missing directory for
--state-fileor--reportwas found only after the first page was encrypted, or after the count pass. - No test failed when a
--limitrun moved the resume position past unsent objects, or when the report was overwritten instead of appended.
Minor, fixed: overflow of the backoff exponent after 8.5 hours of 429 answers; Retry-After with inf or an HTTP date; an unused console; pytest.raises without match; docs on skipped objects on resume and on exit code 1; no upper bound for --concurrency; a zero latency sample; --priority not using the SDK enum.
Minor, left as they are:
- The latency that drives the concurrency limit includes pacing and retry waits, so one 429 with
Retry-After: 20lowered the limit twice. - "Done before the error" counts batches of a page that later fails. The state file is correct.
- The report is flushed before the state is saved, so a kill between the two adds the same report lines again on resume.
Security review
The built-in security-review skill ran once, by an agent other than the author, on d5161c5f: the code of the final commit 1b8c6d87 without the paging position change (about 20 lines in migration.py). Its diff was this change; the agent did the analysis by hand.
0 high, 0 medium, 1 low (4 findings dropped as below the skill's confidence rules or hardening only).
Low, left as it is: text from the server is printed after Rich escapes its brackets, but escape characters are kept, so a compromised server could change the terminal title or clear the screen. Other infrahubctl commands print server text the same way.
Checklist
- Tests added/updated
- Changelog entry added
- External docs updated (generated CLI page)
- Internal .md docs updated (
dev/knowledge/cli-architecture.md)