feat: back up and restore external databases on Kubernetes
Summary
Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This branch gives those deployments the same infrahub-backup create / restore round trip everyone else has, with no manual steps on the database hosts beyond documented one-time prerequisites, and leaves fully-internal deployments byte-for-byte unchanged.
Key Changes
- Operators with external databases can back up and restore with the commands they already use. The tool creates a short-lived pod in the namespace as the vendor's own "backup client", runs
neo4j-admin database backup --from=…orpg_dumpthere, and streams the artifact through the existing pipeline. Artifacts are byte-compatible with internal ones and restore by the same code. - Where a database lives is decided from positive evidence. A deployment query that fails is an error naming the permission, never a silent switch to the external path, so a permission problem cannot create workloads or pull images.
- Every condition that would yield an unusable artifact is caught before data moves: a utility older than the server, an external Neo4j Community database, missing workload permissions, a restore artifact the server cannot read.
- Remote Neo4j restore seeds from an object-store URI and is confirmed, not assumed. The tool polls the database until it is online and fails the run otherwise; "accepted" is not "restored".
- Destructive restore into an unmanaged database needs its own authorisation (
--allow-external-restore/INFRAHUB_ALLOW_EXTERNAL_RESTORE), distinct from--forceand from scheduled restore. - Nothing outlives the run. The transient pod carries its own deadline, its credential Secret is garbage-collected with it, and a stray reaper removes leftovers from earlier runs. Connections are encrypted and verified by default with a single operator opt-out.
- Incomplete captures are never retained, completeness is read from the backup command's output rather than its exit status, and metadata additions are additive, so older releases keep restoring new artifacts.
- Kubernetes resource ownership is decided in one place from label evidence and declared release prefixes, replacing the substring matches that could scale another product's
redis-cacheto zero. - The linter runs again.
make lintwas missing the build tag the Makefile defines and passes to both test and vet,make dev-setuppinned a v1 that cannot read this repo's v2 config, and a linter built by an older Go than the one in use fails every package on typecheck. All three are fixed and documented.
Related Context
- Spec, research, contracts and task ledger:
specs/archive/007-external-database-backup/(archived with its extraction record). User Story 1 (P1 round trip) and User Story 2 (P2 diagnosable refusals) are delivered; User Story 3 (operator-host execution) stays deferred as specified. - Decisions recorded as ADRs 0008–0016 under
dev/adr/. - First real Kubernetes round trip (minikube, chart 4.33.2 with
neo4j.enabled=false, Neo4j 2025.10.1 Enterprise, MinIO as the artifact store) passed end to end; its residual findings are ledgered as T156–T161 in the archivedtasks.md.
Documentation Updates
- Operator docs:
docs/docs/backup/external-databases.mdx(prerequisites, flags, authorisation, troubleshooting),docs/docs/backup/external-database-direct-to-cloud.mdx, and the flags indocs/docs/reference/configuration.mdx. - Developer docs:
dev/knowledge/external-database-backup.md,dev/knowledge/container-image-user-semantics.md,dev/guidelines/kubernetes-backend.md,dev/guidelines/implementation-verification.md, ADRs 0008–0016, andAGENTS.mdsections for the new files.
Test Plan
make test(orgo test -tags untested_go_version ./src/...): unit coverage for endpoint resolution, both version schemes, metadata compatibility, completeness derivation, ownership decisions, transient-workload manifests and lifecycle. All green.make linton Linux, orgolangci-lint run --build-tags untested_go_version ./src/...on macOS, wheretools/neo4jwatchdogcannot build. Reports 0 issues. Re-runmake dev-setupfirst if the Go toolchain has moved since the linter was installed.- End-to-end:
tests/e2e/test_k8s_external_neo4j.pyandtests/e2e/test_k8s_external_both.pyagainst a cluster with the database(s) outside the Infrahub namespace;tests/e2e/test_docker_external_compat.pyfor the unchanged Docker path. - Manual check on a fully-internal deployment:
infrahub-backup createandrestorebehave and lay out artifacts exactly as before (FR-015).
🤖 Generated with Claude Code
Assisted-by: speckit-opsmill-auto unversioned
Assisted-by: speckit-opsmill-implement unversioned
Assisted-by: speckit-opsmill-extract unversioned
Assisted-by: opsmill-dev-commit 0.1.0
Assisted-by: opsmill-dev-pr 0.2.0
Summary by cubic
Adds backup and restore for Kubernetes deployments whose Neo4j Enterprise or PostgreSQL database runs outside the cluster. infrahub-backup create and restore previously could not reach those databases; they now use the vendor's backup client in a transient in-namespace Job and stream artifacts through the existing pipeline. Fully internal deployments are unchanged.
New Features
- The transient Job runs the
infrahub-backupimage, which shipsneo4j-adminand the PostgreSQL client, so no vendor image pull is needed; when the tool itself runs as a pod in the namespace, direct mode runs the tools in-process and creates no Job or Secret. - Credentials reuse the deployment's own Secret keys when explicit
secretKeyRefreferences exist, falling back to a run-owned Secret otherwise. - Restoring an external Neo4j database stages artifacts in object storage and polls the database to confirm recovery; destructive external restores need separate authorization, external TLS is verified by default, and the Job's time-to-live removes the pod and its credential Secret after a killed run.
- Adds operator documentation for prerequisites, configuration, troubleshooting, and direct-to-cloud Neo4j backups.
Bug Fixes
- Database location resolution fails on Kubernetes query, permission, or workload-listing errors instead of assuming an external database; each database reports its location independently.
- Restores gate only the databases their archive carries; Docker restores wait for freshly healthy databases before starting dependent services; external capture streams report truncation and timeout failures instead of retaining incomplete artifacts.
- Credential reuse ignores
envFrom-backed variables, and the PostgreSQL keyword/value connection form is parsed like libpq. - With
--k8s-namespaceor--projectset, a refused or missing namespace fails the run with kubectl's message instead of falling through to a Docker Compose project. - Direct-mode runs use an absolute scratch path, cancel the child's process group, and bound copies by
--external-db-timeout.