fix(backend): keep the Activities page fast with a year of activity log [INFP-507]

Draft
SA
6d ago
+2681-154

Superseded by #11289. The six INFP-507 PRs are consolidated there into one commit targeting develop, with the complete design, code-flow diagrams and contract-based review guide. This PR is closed to keep one active review; its branch and previous discussions are preserved.

Stacked PR 3/6 for INFP-507 (task history and activity log retention), in GitHub stack #10873 on develop. Review and merge bottom-up.

  1. #10862: retention settings applied to Prefect
  2. #10869: task history cleanup command, and the upgrade's task history hint
  3. This PR: Activities page queries
  4. #10877: guard for Prefect's event types on the full stack
  5. #10878: infrahub tasks background-services
  6. #10879: documentation

Carries opsmill/infrahub#10379 (count only when requested, per-query plan), cherry-picked with -x as its two original commits. Once #10379 merges, a rebase of this stack drops them.

Why

With a year of activity log, the Activities page takes 3 to 6 s to load and the sub-activities, branch and account filters take seconds to tens of seconds, because they match labels that have no index and count every matching event. The activity log retention (PR 4) must not be raised before the page stays fast. Design doc: Task history and Activity log retention (D6); spec set in dev/specs/infp-507-task-history-retention/.

Goal: the same events as today, read through the indexed resource ID that Prefect already stores, newest first, without a count unless asked, paged by time.

Closes part of INFP-507.

What changed

Behavioural changes:

  • Filters by ID. The account, branch, sub-activities (parent) and branch merged/rebased/migrated filters match Prefect's indexed event_resources.resource_id instead of labels, as mapped in data-model.md. A branch name resolves to its ID: the current branches read from the database in one query for all the names (a branch whose deletion has started included, since its deletion event is not sent yet), a deleted one through the newest infrahub.branch.deleted event; a name that matches neither returns an empty page without querying. A name reused after a delete shows only the current branch's events.
  • Newest-first windows. The task manager reads the newest hour first, then 1 day, 7 days, 30 days, then the whole activity log retention, counted back from until (or now), and stops at the first window that holds offset + limit matches. The oldest bound is the activity log retention (Prefect's server.events.retention_period, which PR 1 sets from activity_log) instead of a fixed 180 days, so a longer retention shows older events.
  • Count only when selected (from #10379): the server counts matching events only when the GraphQL query selects count; the Activities page does not. #10379 also forces a per-execution plan for the event query, so it no longer flips to a quadratic generic plan after five runs.
  • Paging by time. "Load more" on the Activities page continues from the oldest event shown (until, no offset) and drops repeats by ID, so events recorded while scrolling are neither repeated nor skipped. The proposed-change timeline, which reads oldest first, pages with since.

Implementation notes:

  • No change to Prefect's database (no index, table or extension) and no change to the public GraphQL schema.
  • tests/helpers/event_filters.py::LabelEventFilter keeps today's label-based filter as the reference the equivalence tests compare against.
  • The equivalence test stores events up to days old on the Prefect test server that the telemetry tests share. At some times of day one of them fell inside the telemetry test's previous-UTC-day window (CI: branches_deleted 4 instead of 3). The telemetry test now compares with a count taken before it seeds; with the previous version, the pair fails at 13:09 UTC and passes with the fix.

Open question for the tech owner: the "Primary Node" filter

Three event types carry their primary node only as a label, because their resource ID is built from the branch name: infrahub.branch.merged and infrahub.branch.deleted (the proposed change they belong to) and the group auto-created event (the group). An ID-only filter would drop them from the proposed-change and group timelines. So the node filter matches the indexed resource ID only when the request lists event types and none of those three; otherwise it keeps today's label match, and the node-filter speed-up in the design doc does not reach that case.

[CLAUDE RECOMMENDED – based on the task manager owning its event query] Match resource_id in the node IDs, OR the node label restricted to those three event types, in the task manager's query. That keeps the same events and uses the index for everything else. Not done here because it changes the filter's shape and needs the owner's agreement.

Known limitations

  • Prefect orders events by occurred only. If 40 or more events share one timestamp to the microsecond, "load more" cannot move past them.
  • The GraphQL since argument description still says it defaults to 180 days back. Fixing it changes schema/schema.graphql, so it is left for an explicit decision.

How to review

  1. backend/infrahub/task_manager/event/models.py (filter construction), graphql/queries/event.py (branch-name resolution, count flag), then prefect_server/database.py (windows) and events.py/models.py.
  2. frontend/app/src/entities/events/ui/queries/get-events.query.ts.
  3. Tests:
    • tests/component/task_manager/test_event_filter_equivalence.py: 24 cases, each asserting the exact event IDs and the same pages as the label filter at two offsets.
    • test_event_query_windows.py: 11 cases; newest-first windows equal one full read.
    • tests/unit/task_manager/event/test_models.py.
    • tests/component/graphql/queries/test_event.py.
    • The Vitest file.
    • tests/e2e/activities/test_global_activities.py: loads more while new events are created and checks that no activity appears twice. It fails on the previous frontend with 5 repeated activities.

How to test

uv run pytest backend/tests/unit/task_manager/event/test_models.py -n 0
uv run pytest backend/tests/component/task_manager/test_event_filter_equivalence.py backend/tests/component/task_manager/test_event_query_windows.py backend/tests/component/graphql/queries/test_event.py -n 0
uv run pytest -c tests/e2e/pytest.ini tests/e2e/activities/test_global_activities.py

Impact & rollout

  • Backward compatibility: same events per filter (asserted by the equivalence tests). API consumers that select count still get it; it is only no longer computed when nobody asks for it.
  • Performance: speed with a year of activity log on Postgres 14 and 18 is measured by the private performance tests (TestActivityLog in opsmill/infrahub-private-tests), run separately; results go on INFP-507.
  • Deployment notes: none beyond the stack.

Validated locally: unit, component, Vitest and E2E (against an image built from this branch); /pre-ci.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (PR 6 of this stack)
  • Internal .md docs updated (dev/knowledge/backend/events.md, from #10379; more in PR 6)
  • I have reviewed AI generated content

🤖 Generated with Claude Code