feat(backend): add task manager retention settings [INFP-507]

Draft
SA
6d ago
+3462-14

Superseded by #11289. The six INFP-507 PRs are consolidated there into one commit targeting develop, with the complete design, code-flow diagrams and contract-based review guide. This PR is closed to keep one active review; its branch and previous discussions are preserved.

Stacked PR 1/6 for INFP-507 (task history and activity log retention), in GitHub stack #10873 on develop. Review and merge bottom-up; each PR targets the one below it.

  1. This PR: retention settings applied to Prefect
  2. #10869: task history cleanup command, and the upgrade's task history hint
  3. #10876: Activities page queries
  4. #10877: guard for Prefect's event types on the full stack
  5. #10878: infrahub tasks background-services
  6. #10879: documentation

Ship with #10869 in the same release. On its own, this PR switches on Prefect's hourly deletion, which works through a large backlog slowly and gives no disk space back; #10869 adds the one-off cleanup command, its table rewrite, and the upgrade's hint that names that command.

Why

The task manager never deletes finished task runs, so task history grows until the disk fills, and the activity log is fixed at Prefect's 7 days with no Infrahub setting. Design doc: Task history and Activity log retention; spec set in dev/specs/infp-507-task-history-retention/.

Goal: one Infrahub setting per kind of record, translated into the Prefect settings that enforce it, so Prefect's built-in hourly cleanup bounds task history from this PR on.

Non-goals here: deleting the existing backlog and giving disk space back (PR 2), the Activities query changes (PR 3).

Closes part of INFP-507.

How retention works

Infrahub defines the retention policy; Prefect's background services enforce it. Task history and activity are kept independently, while Infrahub and Prefect events share the same storage.

flowchart TB
    CONFIG["Infrahub retention settings<br/>Task history · Activity log · Prefect events"]
    TYPES["Infrahub-maintained list<br/>of Prefect event types"]

    subgraph TM["Task manager"]
        APPLY["At startup<br/>Translate Infrahub settings into Prefect settings"]

        subgraph SERVICES["Prefect background services — every hour"]
            TASKS["Task-history cleanup<br/>Remove finished runs older than task_history<br/>together with their logs and artifacts"]
            EVENTS["Event cleanup<br/>Keep Infrahub events for activity_log<br/>Keep listed Prefect events for prefect_own_events"]
        end

        APPLY --> TASKS
        APPLY --> EVENTS
    end

    subgraph DB["Prefect database"]
        RUNS[("Task history<br/>Runs, states, logs and artifacts")]
        LOG[("Shared event storage<br/>Infrahub activity + Prefect internal events")]
    end

    CONFIG --> APPLY
    TYPES --> APPLY
    TASKS -->|"Delete expired task history"| RUNS
    EVENTS -->|"Delete expired events and related resources"| LOG

Increasing activity-log retention can preserve a year of Infrahub activity without also keeping a year of Prefect's internal events. When background services run in their own deployment, #10878 applies the same settings before starting them.

The operator cleanup in #10869 is a separate recovery step for existing instances: clear the old task-history backlog and recover disk space after an upgrade or retention reduction. The upgrade prints guidance; ongoing retention is handled automatically by the flow above.

What changed

Behavioural changes:

  • New settings task_manager.retention.task_history (30 days), activity_log (7 days, unchanged from today) and prefect_own_events (7 days); env vars INFRAHUB_TASK_MANAGER_RETENTION_*; 30d or ISO 8601; 1 to 36,500 days, or the task manager refuses to start.
  • Finished task runs older than 30 days are now deleted every hour with their logs and artifacts (Prefect's flow-run vacuum is switched on). Previously they were never deleted.
  • Prefect's own events (103 event types from Prefect 3.8.6: run and task-run states, heartbeats, workers, work pools, automations...) follow prefect_own_events; Infrahub events follow activity_log. With the defaults both are 7 days, as today.
  • A PREFECT_* variable already set (under any name Prefect accepts, including legacy aliases) takes precedence, with one warning per variable. The effective values are logged at startup.

Implementation notes:

  • backend/infrahub/prefect_server/retention.py builds the four Prefect variables; create_infrahub_prefect now loads the Infrahub configuration in every mode (it only did in distributed mode), applies them, then refreshes Prefect's global settings context (Prefect reads its settings once at import).
  • The integration-test Prefect container runs the stock Prefect image without the infrahub package, so it now uses create_prefect_app_with_infrahub_routes, which skips the configuration step.
  • The Compose task-manager service does not merge the shared Infrahub anchor, so the three variables are declared on it explicitly (root docker-compose.yml and the dev deps files).
  • tasks/docs.py: the configuration reference showed bare field names for a nested settings model with its own env prefix; it now shows the INFRAHUB_TASK_MANAGER_RETENTION_* names.

What stayed the same: no change to Prefect's database schema; Infrahub events are untouched; the component-test harness still turns the vacuum off (its preset variable wins by design).

How to review

  1. backend/infrahub/config.py (settings and validation), then prefect_server/retention.py and prefect_server/app.py.
  2. Tests: tests/unit/config/test_task_manager_retention_settings.py, tests/unit/prefect_server/ (one test feeds the derived env into Prefect's own settings model, so a format mistake fails CI), tests/component/task_manager/test_event_retention.py (runs Prefect's event vacuum with a 365-day activity log and 7-day own events).
  3. dev/specs/infp-507-task-history-retention/ is the spec set for the whole stack.

Worth extra scrutiny: the event-type list. Overrides match event names exactly, and an unlisted Prefect type is kept for the activity log retention. A unit test covers every built-in state name; PR 4 adds a full-stack guard on a real workload.

How to test

uv run pytest backend/tests/unit/config/test_task_manager_retention_settings.py backend/tests/unit/prefect_server/ -n 0
uv run pytest backend/tests/component/task_manager/test_event_retention.py -n 0

Manual: start the task manager with INFRAHUB_TASK_MANAGER_RETENTION_TASK_HISTORY=1d and look for the Task manager retention: lines; with 0d it refuses to start.

Impact & rollout

  • Backward compatibility: irreversible deletion of finished runs older than the retention starts within an hour of the upgrade. Operators who need older runs set a longer task_history first (changelog says so).
  • Performance: the first hourly passes delete the backlog in Prefect's batches; PR 2 adds the faster one-off cleanup, which operators run once after the upgrade; the upgrade prints the command.
  • Config/env changes: three new settings; PREFECT_* precedence documented.
  • Deployment notes: the Helm chart needs the same three variables on the task-manager deployment (infrahub-helm PR follows the stack).

Validated locally: unit and component tests, ruff, mypy, docs.validate, validate-dockercomposeenv; reviewed with cubic (two passes, findings addressed). Release evidence on restored backups comes from opsmill/infrahub-private-tests, run separately.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (PR 6 of this stack)
  • Internal .md docs updated (PR 6 of this stack)
  • I have reviewed AI generated content

🤖 Generated with Claude Code