feat(collect): report task manager database storage in bundles

FA
5d ago
+7428-118

Why

A bundle from infrahub-collect create does not show how much disk space the task manager's PostgreSQL database (task-manager-db) uses, or which records use it. Support then has to send the customer SQL queries to run (#181).

With this change, support can answer these questions from the bundle alone (PRD, spec):

  1. How big is the database?
  2. Which tables use the space?
  3. Is that space live rows, or deleted rows that VACUUM has not reclaimed yet?
  4. Within the tables that hold flow runs, task runs, states, logs, events and artifacts, which kinds of records use the space?

Every bundle answers questions 1 to 3. Question 4 needs the new --include-task-manager-storage flag, because answering it reads every row of those tables.

Out of scope, as listed in the spec:

  • External PostgreSQL servers, which will follow #172.
  • Byte sizes for each group of records in the breakdown.
  • Free space on the database volume, and growth over time.
  • Interpreting the figures, or running VACUUM or a flush.
  • Neo4j storage figures, and Prefect 2.x deployments.

Closes #181

What changed

Storage figures in every bundle

  • A new collector, task-manager-db-storage, writes task-manager-db/storage.json:
    • the database size, and when PostgreSQL last reset its statistics;
    • for each table, its data, index and TOAST sizes, its live and dead row estimates, and its last vacuum times.
  • The query has a 60-second limit.

Storage breakdown with --include-task-manager-storage

  • The flag can also be set with INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.
  • A second collector writes task-manager-db/storage-breakdown.json. It covers seven Prefect tables: events, flow_run, task_run, flow_run_state, task_run_state, log and artifact.
  • For each table, it gives the exact row count and the oldest and newest timestamps of each group of records. The groups are, for example, flow name and state type for flow runs, or flow name and log level for logs.
  • Each table's query has a 5-minute limit. A table that reaches the limit is recorded as failed, and the other tables are still collected. A table that the deployed Prefect version does not have is recorded as absent.

Failures

  • A failure never stops the bundle. The manifest records the reason, for example timed out after 60s, login rejected: ..., psql missing: ... or not reachable: ....
  • The diagnostics go to storage.err.txt or storage-breakdown.err.txt, with the password masked.
  • Both collectors are skipped when the deployment has no task-manager-db service.

Change to infrahub-backup

  • infrahub-backup reads the user, password and database only from the Prefect connection URL itself.
  • A part that the URL leaves out takes the default. Before this change, it took a value from PGUSER, PGPASSWORD, PGDATABASE, ~/.pgpass or the operating-system user of the machine that runs the tool.
  • The warning about a URL that does not parse now names the part that failed, without quoting the URL.

Implementation notes

  • Every query goes through one runner, runTaskManagerDBQuery, which runs psql inside task-manager-db (ADR 0008):
    • The session is read-only.
    • PostgreSQL enforces the time limit (statement_timeout), so a query stops on the server even after the tool stops waiting for it.
    • The breakdown queries turn off hash joins, merge joins and parallel workers, so they write no temporary files to a volume that may be nearly full.
  • The password reaches psql on stdin, through a short sh script, so it is in no argument of the docker, kubectl, sh or psql process.
  • The tool looks for the login in this order:
    1. a Prefect connection URL in the operator's environment;
    2. a Prefect connection URL in the task-manager container;
    3. POSTGRES_USER or POSTGRES_PASSWORD in task-manager-db;
    4. the defaults of the backup tool.
  • The Prefect connection URL is parsed with url.Parse instead of pgx.ParseConfig, so github.com/jackc/pgx/v5 is no longer a dependency. The Nix vendorHash is updated.
  • On a CloudNativePG cluster, the queries run in the primary pod.

What stayed the same

  • The backup archive format and infrahub-backup create are unchanged. infrahub-backup create still passes the database password to pg_dump as a docker or kubectl argument.
  • The other collectors and the existing infrahub-collect flags are unchanged.

Extraction of the spec into dev/

The last commit, 2380139, runs speckit-opsmill-extract on the spec, so the PR lands with the knowledge extracted. The PR had already written ADR 0008, the storage section of the knowledge page and the timeout-wrapping exception of the guidelines; the commit adds what was still missing:

  • ADR 0009: PostgreSQL is reached only through psql, pg_dump and pg_restore inside task-manager-db, over TCP to localhost, with the alternatives that research R1 rejected (a Go client, Unix-socket peer authentication, the Prefect server container).
  • dev/knowledge/infrahub-collect.md: the file contracts of storage.json and storage-breakdown.json, the manifest reason for each outcome, the five error kinds and the progress lines, the privileges the queries need, and a gotcha: the official postgres image trusts localhost, so a wrong password still logs in on Docker Compose.
  • dev/guidelines/collectors.md: one JSON value per query, the breakdown list as data, and how to test a database time limit and a change to the password path end to end.
  • The spec moved to specs/archive/006-taskmgr-db-storage/ with its extraction record. The references to it in ADR 0008, two Go comments and tests/helpers/bundle.py point at the archive.

Suggested review order

  1. ADR 0008 and the spec, then ADR 0009.
  2. src/internal/app/collect_taskmanager_db.go: credential lookup, query runner, error classification and masking.
  3. src/internal/app/collect_storage.go: the SQL and the two collectors.
  4. src/internal/app/app_config.go: the Prefect URL parsing, which infrahub-backup also uses.
  5. ExecSeparateInputContext in environment_docker.go and environment_kubernetes.go.
  6. The unit tests, then tests/e2e/test_docker_collect.py and tests/e2e/test_k8s_collect.py.

The files under specs/archive/006-taskmgr-db-storage/ are planning records. They can be skimmed. The last commit extracted the spec into dev/ and archived it (see "Extraction of the spec into dev/" above).

How to review

These parts need the most attention:

  • The sh script that reads the password from stdin. On Kubernetes it depends on kubectl exec -i. This path has unit tests only: no end-to-end run has covered it yet (see "How to test").
  • Locks. Each breakdown query holds an ACCESS SHARE lock on the table it reads and on the tables it joins, for up to 5 minutes. During a task manager upgrade, a schema change on one of these tables waits until the query ends. The user documentation warns about this.

These review findings are not addressed in this PR:

  • The collect tool has its own credential lookup, with a different fallback order from the backup tool's. When the Prefect server is down, infrahub-collect can log in to task-manager-db while infrahub-backup, and --include-backup, fall back to the defaults.
  • On Kubernetes, the pod name is looked up once, and every query of the breakdown reuses it. If CloudNativePG moves the primary to another pod during the breakdown, which can take up to 35 minutes, the remaining queries go to the old pod.
  • The tool uses the user, password and database of the Prefect URL, but not its host and port. It always connects to localhost:5432 inside task-manager-db.
  • The breakdown does not include Prefect 3's event_resources table.

The implementation report lists the other deferred findings.

How to test

make build-watchdog   # once per worktree
make test
make vet
GOTOOLCHAIN=go1.25.0 go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.7.2 run
uv run pytest -v -m docker tests/e2e/test_docker_collect.py
uv run pytest -v -m k8s tests/e2e/test_k8s_collect.py

Results:

  • make test, make vet and golangci-lint (0 issues) pass on 419cd9f, and so do rumdl, Vale and nix build.
  • The Docker and Kubernetes end-to-end tests passed on 6f21171, before the last two commits. The last two commits change how the password reaches psql, and CI runs both end-to-end suites on this PR.
  • The stdin path was checked through the real Docker backend against a postgres:18-alpine container that requires a password on localhost:
    • A password with spaces, quotes, @ and $ logs in.
    • The session is read-only and named infrahub-collect.
    • No host process shows the password in its arguments while a query runs.
    • A wrong password is reported as login rejected.
  • The CloudNativePG primary-pod check is recorded in t046-cnpg-check.md. It ran before the rebase on main and was not run again.
  • Both collectors and infrahub-backup create were run against the least-privilege PostgreSQL setup of the Infrahub production deployment documentation ("Least-privilege service credentials"): a non-superuser prefect role that owns the prefect database and the public schema, on postgres:18-alpine started with --auth-host=scram-sha-256, with the Infrahub 1.12.0 task manager connected as that role and a percent-encoded password holding @, a space, /, $ and #. Both storage collectors were success, PostgreSQL's connection log showed every session as user=prefect database=prefect application_name=infrahub-collect, pg_dump connected as prefect as well, and a wrong password in the operator's environment was reported as login rejected.
  • The extraction commit 2380139 changes only Markdown, two Go comments and comments in tests/helpers/bundle.py. go vet, the unit tests, rumdl and ruff pass on it.

Impact & rollout

  • Backward compatibility:
    • Every bundle holds a new file, task-manager-db/storage.json, and the manifest lists two new collectors.
    • The backup format is unchanged.
    • infrahub-backup no longer takes missing parts of the Prefect URL from PG* variables or ~/.pgpass.
  • Both backends: Docker Compose and Kubernetes behave the same. On Kubernetes, the queries run in the primary pod of a CloudNativePG cluster.
  • Config/env changes:
    • New flag --include-task-manager-storage, with the variable INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE.
    • INFRAHUB_TASK_MANAGER_DB_QUERY_TIMEOUT lowers the query limits for the end-to-end tests only, and is not documented for operators.
  • Deployment notes: task-manager-db needs sh and psql, which the official postgres, Bitnami and CloudNativePG images include. Without them, the manifest records the collector as failed, and the bundle is still produced.

Checklist

  • Tests added/updated
  • Changelog entry added (changelog/181.added.md, changelog/181.changed.md)
  • External docs updated (docs/docs/collect/, docs/docs/reference/)
  • Internal .md docs updated (dev/adr/0008-*, dev/adr/0009-*, dev/guidelines/collectors.md, dev/knowledge/infrahub-collect.md, AGENTS.md)
  • I have reviewed AI generated content

šŸ¤– Generated with Claude Code


Summary by cubic

Troubleshooting bundles now include the storage use of the task manager database, so support can answer how much disk the database uses and which records fill it without sending the customer SQL queries (closes #181).

Task manager database storage collectors

  • Every bundle now holds task-manager-db/storage.json: database and per-table sizes, live and dead row estimates, and last vacuum times.
  • --include-task-manager-storage (or INFRAHUB_INCLUDE_TASK_MANAGER_STORAGE) adds task-manager-db/storage-breakdown.json: exact row counts and oldest/newest timestamps per record group for the flow run, task run, state, log, event and artifact tables.
  • Queries run read-only in task-manager-db with time limits PostgreSQL enforces; a failed or missing table is recorded in the manifest without stopping the bundle, and the password reaches psql on stdin, not in a process argument.
  • Do not request the breakdown during a task manager upgrade: each query holds an access-share lock on the tables it reads and joins for up to 5 minutes.

infrahub-backup credential lookup

  • The backup tool now reads the task manager database user, password and database only from the Prefect connection URL, taking defaults for missing parts; it no longer falls back to PG* variables, ~/.pgpass or the OS user, and pgx is replaced with net/url parsing.