docs: document task history and activity log retention [INFP-507]

Draft
SA
6d ago
+1142-64

Superseded by #11289. The six INFP-507 PRs are consolidated there into one commit targeting develop, with the complete design, code-flow diagrams and contract-based review guide. This PR is closed to keep one active review; its branch and previous discussions are preserved.

Stacked PR 6/6 for INFP-507 (task history and activity log retention), in GitHub stack #10873 on develop. Review and merge bottom-up.

  1. #10862: retention settings applied to Prefect
  2. #10869: task history cleanup command, and the upgrade's task history hint
  3. #10876: Activities page queries
  4. #10877: guard for Prefect's event types on the full stack
  5. #10878: infrahub tasks background-services
  6. This PR: documentation

Why

Operators need to know before they upgrade that finished task runs older than the task history retention are deleted for good, which instances need the cleanup step after the upgrade, how long it takes, and how much free disk the rewrite needs. The internal docs and ADR-0002 still assume Prefect's default retention covers the audit trail.

The design doc's Design changes for implementation list (8 and 11 Oct) asked this PR to describe the cleanup step as a step after the upgrade for existing instances with a lot of task history, the same for Compose and Helm with no need to stop anything; to add a tip about short pauses on very large instances; to say it is the same step after lowering a retention; and to drop "the event tables have no rewrite". All four are done.

Closes part of INFP-507.

What changed

Customer docs (docs/docs/):

  • Upgrade guides (maintain-upgrade/upgrade/overview.mdx, community.mdx, enterprise.mdx):
    • the upgrade deletes no task runs and prints how much disk space the task history takes, how many task runs are older than the retention (up to 100,000), and the command that gets the space back
    • a section "Clean up old task history after the upgrade": who needs the step (existing instances with a lot of task history; new and small instances do not), the same command for Docker Compose and Helm with the instance running, and what each line of its output means
    • a tip that on very large instances tasks may pause for a few seconds while tables are rewritten, with the benchmark figures; for the activity log tables the pause was not measured
    • deletion is irreversible, so set a longer task_history before upgrading
    • the benchmark durations: about 8.5 minutes for 25 GB of task history, about 2 h 20 min for 100 GB
    • the free disk the rewrite needs
    • the same step after lowering the task history or the activity log retention
    • the Community and Enterprise guides point to the section after their upgrade commands, for Docker and Helm
  • run-observe/tasks.mdx:
    • the task history retention
    • the precedence of PREFECT_* variables, legacy aliases included
    • flush flow-runs [--rewrite], which rewrites each task history and activity log table that is mostly free space, with the instance running
    • flush stale-runs and its limit: it does not catch PENDING runs that never started
    • background-services
  • run-observe/activity-log.mdx:
    • the activity log and own-event retention
    • the startup warning about Prefect event types missing from Infrahub's list, what it costs in disk, and where to report the types
    • sizing at 5.7 to 8.1 GiB per million stored events, with an example of scaling a week of events to a planned retention
    • removed the claim that queries without since stop at 180 days
    • after lowering the activity log retention, flush flow-runs --rewrite gets the disk space back
  • Generated CLI reference for infrahub tasks (tasks/docs.py::CLI_COMMANDS, sidebars.ts). The generated page exposed a wrong description of infrahub tasks execute, fixed in its docstring.

Internal docs:

  • New dev/knowledge/backend/task-manager-retention.md with behaviour tables: settings and precedence, hourly cleanup, the cleanup job with its per-table rewrite and the upgrade's hint, stuck runs, the Prefect event-type list. Linked from events.md, async-tasks.md, backend/AGENTS.md and dev/README.md. async-tasks.md was already long, so the detail lives in its own page.
  • events.md: retention, the event-type list, the Activities queries.
  • dev/adr/0002-events-system.md: an amendment records that the audit trail now follows task_manager.retention.activity_log, and is no longer Prefect's default of 7 days. Status and decision unchanged.
  • dev/specs/telemetry-collection-infp-589/: 90 days of task history corrected to the 30-day default.

Open points for reviewers

  • Helm.
    • The cleanup step on Helm is kubectl exec into the running server pod, so it needs no change to the chart; the chart's upgrade hook needs no extra option. The chart change for the background services and the retention values is opsmill/infrahub-helm#100, a draft until the chart's images move to Infrahub 1.13.
    • The Enterprise sizing presets run the background services in their own pod with prefect server services start, so the hourly deletion does not run there until that chart PR switches the pod to infrahub tasks background-services. The chart PR must ship in the same release.
  • Telemetry spec. It assumes the task history reaches back well beyond 24 hours, but the retention can now be set as low as 1 day. Not changed here.
  • Sizing example. It is arithmetic on the measured 5.7 to 8.1 GiB per million events, not a separate measurement.

Quickstart Part 1 steps 5 and 6 after the design changes (11 Oct)

A new dev stack from the top of this stack, with 30,000 finished runs that ended 35 days ago:

  • Step 5 as the Compose guide gives it (down, then run --rm server infrahub upgrade): no cleanup step, and Task history takes 26.6 MB, with no runs older than the retention. To get the space back, run: infrahub tasks flush flow-runs --rewrite. The task manager's own cleanup had deleted the 30,000 runs when it started, before step 5.
  • Step 6 with the instance running: Deleted 0 runs, Task history and activity log tables: 29.9 MB before, 4.9 MB after, Rewrote flow_run, flow_run_state, task_run, artifact, and the other four tables left as mostly live. flow_run went from 24 MB to 816 kB, in 8 s.
  • The same two steps inside the running instance, as on Helm, after 30,000 more old runs: Task history takes 27.3 MB, with 30,000 runs older than the retention., then Deleted 30000 runs, 31.4 MB before, 6.5 MB after, Rewrote flow_run, artifact.

Quickstart results on a local Compose stack (4 to 8 Oct, before the design changes)

dev/specs/infp-507-task-history-retention/quickstart.md, run on the dev Compose stack with an image built from the top of this stack. Steps 5 and 6 of Part 1 changed on 11 Oct and ran again, see above.

  • Part 1, task history:
    • Steps 1 to 6 and 9 passed.
    • Step 5 (infrahub upgrade with old runs) failed on the first run. Run as the Compose guide gives it (down, then run server infrahub upgrade), the task manager starts and Prefect's own cleanup deletes the old runs before the upgrade's step starts its job, so the job never rewrote the tables (166.5 MB before, 166.5 MB after). Fixed in #10869: the rewrite now depends on free space in flow_run. The second run printed 175.0 MB before, 1.6 MB after / Task history tables rewritten.
    • Step 7 was not run, because no task manager from the previous release was at hand; unit tests cover the 404 path.
    • Step 8 is covered by the scale-out stack check of #10878.
  • Part 2, Activities: the filter equivalence test and the Activities E2E test pass. The Postgres statement log shows the count query only for a GraphQL query that selects count.
  • Part 3, activity log retention: an activity log of 365d with own events of 30d gives PREFECT_SERVER_EVENTS_RETENTION_PERIOD=P365D and P30D for 103 event types. Step 1 expects the 7-day default for own events; the overrides follow prefect_own_events, and the unit tests cover the default. Own events of 30 days with an activity log of 7 days give the warning and P7D for 103 event types.
  • Part 4, docs: docs.validate passes.

The same run showed that small task history tables usually count as mostly free, because the upgrade replaces Prefect's scheduled runs, and are rewritten in milliseconds.

The constitution deviation (the unauthenticated cleanup route) and the new GET /infrahub/task-history/size route are in #10869's description for approval. The implementation report is in dev/specs/infp-507-task-history-retention/opsmill-implement-report.md.

How to test

uv run invoke docs.validate
cd docs && npm run build

Validated locally on 11 Oct: Vale and markdownlint on the changed customer docs (0 errors), docs.validate, the docs build, and quickstart Part 1 steps 5 and 6 above. Earlier: /pre-ci and the Compose quickstart run above. The kubectl, Compose and GraphQL examples in the pages were not run.

Checklist

  • Tests added/updated (documentation only)
  • Changelog entry added (the behaviour fragments are in the PRs below)
  • External docs updated
  • Internal .md docs updated
  • I have reviewed AI generated content

šŸ¤– Generated with Claude Code