feat: back up and restore external databases on Kubernetes

Merged
BE
2d ago
+41642-524

Summary

Infrahub deployments on Kubernetes whose Neo4j Enterprise database, PostgreSQL task-manager database, or both run outside the cluster had no backup or recovery path through this toolset: every path executed the vendor utility inside a database container, and there was none. This branch gives those deployments the same infrahub-backup create / restore round trip everyone else has, with no manual steps on the database hosts beyond documented one-time prerequisites, and leaves fully-internal deployments byte-for-byte unchanged.

Key Changes

  • Operators with external databases can back up and restore with the commands they already use. The tool creates a short-lived pod in the namespace as the vendor's own "backup client", runs neo4j-admin database backup --from=… or pg_dump there, and streams the artifact through the existing pipeline. Artifacts are byte-compatible with internal ones and restore by the same code.
  • Where a database lives is decided from positive evidence. A deployment query that fails is an error naming the permission, never a silent switch to the external path, so a permission problem cannot create workloads or pull images.
  • Every condition that would yield an unusable artifact is caught before data moves: a utility older than the server, an external Neo4j Community database, missing workload permissions, a restore artifact the server cannot read.
  • Remote Neo4j restore seeds from an object-store URI and is confirmed, not assumed. The tool polls the database until it is online and fails the run otherwise; "accepted" is not "restored".
  • Destructive restore into an unmanaged database needs its own authorisation (--allow-external-restore / INFRAHUB_ALLOW_EXTERNAL_RESTORE), distinct from --force and from scheduled restore.
  • Nothing outlives the run. The transient pod carries its own deadline, its credential Secret is garbage-collected with it, and a stray reaper removes leftovers from earlier runs. Connections are encrypted and verified by default with a single operator opt-out.
  • Incomplete captures are never retained, completeness is read from the backup command's output rather than its exit status, and metadata additions are additive, so older releases keep restoring new artifacts.
  • Kubernetes resource ownership is decided in one place from label evidence and declared release prefixes, replacing the substring matches that could scale another product's redis-cache to zero.
  • The linter runs again. make lint was missing the build tag the Makefile defines and passes to both test and vet, make dev-setup pinned a v1 that cannot read this repo's v2 config, and a linter built by an older Go than the one in use fails every package on typecheck. All three are fixed and documented.

Related Context

  • Spec, research, contracts and task ledger: specs/archive/007-external-database-backup/ (archived with its extraction record). User Story 1 (P1 round trip) and User Story 2 (P2 diagnosable refusals) are delivered; User Story 3 (operator-host execution) stays deferred as specified.
  • Decisions recorded as ADRs 0008–0016 under dev/adr/.
  • First real Kubernetes round trip (minikube, chart 4.33.2 with neo4j.enabled=false, Neo4j 2025.10.1 Enterprise, MinIO as the artifact store) passed end to end; its residual findings are ledgered as T156–T161 in the archived tasks.md.

Documentation Updates

  • Operator docs: docs/docs/backup/external-databases.mdx (prerequisites, flags, authorisation, troubleshooting), docs/docs/backup/external-database-direct-to-cloud.mdx, and the flags in docs/docs/reference/configuration.mdx.
  • Developer docs: dev/knowledge/external-database-backup.md, dev/knowledge/container-image-user-semantics.md, dev/guidelines/kubernetes-backend.md, dev/guidelines/implementation-verification.md, ADRs 0008–0016, and AGENTS.md sections for the new files.

Test Plan

  • make test (or go test -tags untested_go_version ./src/...): unit coverage for endpoint resolution, both version schemes, metadata compatibility, completeness derivation, ownership decisions, transient-workload manifests and lifecycle. All green.
  • make lint on Linux, or golangci-lint run --build-tags untested_go_version ./src/... on macOS, where tools/neo4jwatchdog cannot build. Reports 0 issues. Re-run make dev-setup first if the Go toolchain has moved since the linter was installed.
  • End-to-end: tests/e2e/test_k8s_external_neo4j.py and tests/e2e/test_k8s_external_both.py against a cluster with the database(s) outside the Infrahub namespace; tests/e2e/test_docker_external_compat.py for the unchanged Docker path.
  • Manual check on a fully-internal deployment: infrahub-backup create and restore behave and lay out artifacts exactly as before (FR-015).

🤖 Generated with Claude Code


Assisted-by: speckit-opsmill-auto unversioned
Assisted-by: speckit-opsmill-implement unversioned
Assisted-by: speckit-opsmill-extract unversioned
Assisted-by: opsmill-dev-commit 0.1.0
Assisted-by: opsmill-dev-pr 0.2.0


Summary by cubic

Adds backup and restore for Kubernetes deployments whose Neo4j Enterprise or PostgreSQL database runs outside the cluster. infrahub-backup create and restore previously could not reach those databases; they now use the vendor's backup client in a transient in-namespace Job and stream artifacts through the existing pipeline. Fully internal deployments are unchanged.

New Features

  • The transient Job runs the infrahub-backup image, which ships neo4j-admin and the PostgreSQL client, so no vendor image pull is needed; when the tool itself runs as a pod in the namespace, direct mode runs the tools in-process and creates no Job or Secret.
  • Credentials reuse the deployment's own Secret keys when explicit secretKeyRef references exist, falling back to a run-owned Secret otherwise.
  • Restoring an external Neo4j database stages artifacts in object storage and polls the database to confirm recovery; destructive external restores need separate authorization, external TLS is verified by default, and the Job's time-to-live removes the pod and its credential Secret after a killed run.
  • Adds operator documentation for prerequisites, configuration, troubleshooting, and direct-to-cloud Neo4j backups.

Bug Fixes

  • Database location resolution fails on Kubernetes query, permission, or workload-listing errors instead of assuming an external database; each database reports its location independently.
  • Restores gate only the databases their archive carries; Docker restores wait for freshly healthy databases before starting dependent services; external capture streams report truncation and timeout failures instead of retaining incomplete artifacts.
  • Credential reuse ignores envFrom-backed variables, and the PostgreSQL keyword/value connection form is parsed like libpq.
  • With --k8s-namespace or --project set, a refused or missing namespace fails the run with kubectl's message instead of falling through to a Docker Compose project.
  • Direct-mode runs use an absolute scratch path, cancel the child's process group, and bound copies by --external-db-timeout.