feat: detect upstream movement on read-only repository refs [IFC-3152]

Merged
OG
1mo ago
+2846-52

Why

A read-only repository in Infrahub follows a branch or a tag on someone else's remote. Until now, nothing noticed when that ref moved. Infrahub kept serving whatever it imported the last time somebody triggered an import by hand, and there was no way to find out that the upstream had advanced short of looking.

Goal: notice the movement on a schedule, and make the new commits available to every worker, without changing anything about what Infrahub has actually imported.

That last clause is the whole design constraint, so it's worth being explicit. Detecting movement is not the same as acting on it. A read-only repository's tracked commit is what the rest of the product reads from, and importing is a separate, deliberate operation. So this PR makes the system aware that the remote moved, and leaves the decision to import exactly where it was.

Non-goals:

  • No UI. The "check remote now" button is PR 9 (T072-T075); today the on-demand path is only reachable through the GraphQL mutation.
  • No import, and no change to the tracked commit.
  • No change to read-write repositories, which already sync every minute.

Part of IFC-3101. This is PR 8 of the spec's decomposition and targets the stack's base branch, not develop.

What changed

Behavioral changes

  • Read-only repositories are now checked for upstream movement on a schedule. Each one is checked at most once per INFRAHUB_GIT_READ_ONLY_REFS_CHECK_INTERVAL_MINS (default 15).
  • InfrahubReadOnlyRepositoryCheckRefs now does something. It previously accepted a request, returned a task id, and performed no work. It now runs the same check the schedule does, for the branch the caller is on.
  • A moved ref converges the whole worker pool. Workers fetch the new objects but stay checked out on the commit Infrahub tracks.
  • New setting: INFRAHUB_GIT_READ_ONLY_REFS_CHECK_INTERVAL_MINS, 1 to 1440 minutes, default 15.

How the check works

Per repository, in this order:

  1. Claim it in the cache. If another run holds it, return that run's id and contact nothing — this is what stops the scheduled and on-demand paths piling up on one repository.
  2. List the remote refs (git ls-remote) holding no lock at all. An idle repository costs one listing and transfers nothing.
  3. Return if nothing moved. Most checks end here.
  4. Only if something moved: take the repository lock, fetch, and broadcast RefreshGitFetch pinned to the tracked commit, once per Infrahub branch following that ref.

The lock scope in steps 2 and 4 is deliberate and load-bearing: git applies no network timeout by default, so holding the repository lock across a remote listing would let one unreachable host block that repository's imports indefinitely. The listing changes nothing locally, which is what makes moving it outside the lock safe.

Implementation notes

  • backend/infrahub/git/refs_check/ is five modules, following backend/infrahub/api/admission/ as .agents/rules/backend-component-design.md suggests. gateway.py is the only one that imports GitPython and it translates every git exception into RepositoryError, so the decision logic in checker.py names no library type and is testable without a clone.
  • Scheduling lives in one place. RefsCheckScheduler owns both "is this repository due?" and "make it due again sooner after a failure". A failed repository comes back in 5 minutes rather than on the next tick, so a permanently broken remote can't occupy a concurrency slot every minute at the expense of repositories that can still be checked.
  • The wall-clock bound sits on the listing, not on the whole check. It is applied inside the component rather than as a Prefect task timeout. Prefect cancels through an anyio scope, and a deadline reached inside the convergence would make the lock release raise — and repository.<name> is created with no TTL, so that lock would then be held forever. Giving it a TTL isn't an option either: InfrahubLockRegistry.get caches locks by name, so the first caller's TTL becomes everyone's, and a long import would have its lock expire underneath it.
  • A tag that moved is force-fetched. git refuses to update an existing tag otherwise. The commits the tag used to point at stay readable because each imported commit has a worktree of its own holding it.

What stayed the same

  • No schema changes, no migrations.
  • The GraphQL contract is unchanged — InfrahubReadOnlyRepositoryCheckRefs was published in #10566; only its behaviour is filled in.
  • Read-write repositories are untouched.
  • The tracked commit attribute is never written by any code path in this PR.

Known limitation: repositories added on a branch are not covered

A read-only repository created on a non-default branch is never checked by this feature, on any branch, even after its branch is merged.

build_check_refs_model() only selects branches where the repository's internal_status is active, and the on-demand mutation applies the same rule. But a read-only repository created on a non-default branch is set to staging at creation and never promoted: the staging → active transition in merge_git_repository is gated on the read-write kind, and the read-only branch of that same function copies ref and commit to the destination while leaving internal_status untouched. There is no other assignment to active in the backend.

So such a repository ends up with a real imported commit and a local clone while still reading as staging, and is skipped.

This is pre-existing and not introduced here — this PR is just the first feature to gate on internal_status, which is what makes it visible. Filed as #10730 with the three code paths and a suggested fix. Once that lands, these repositories are picked up by the existing eligibility rule with no change to this code.

Suggested review order

  1. backend/infrahub/git/refs_check/checker.py — the decision logic; check() is the whole flow in one method.
  2. backend/infrahub/git/refs_check/gateway.py — everything that touches git.
  3. backend/infrahub/git/tasks.py — the two flows that build and drive the above.
  4. backend/infrahub/graphql/mutations/repository.py — the on-demand entry point.
  5. Tests, then the regenerated config docs and compose files.

How to review

Where the risk is concentrated:

  • _converge() in checker.py — the only place holding the repository lock, and the only place broadcasting. If the lock scope or the pinned commit is wrong, the blast radius is every worker's copy of that repository.
  • _ref_patterns() / _list_remote_heads() in gateway.py — the ls-remote pattern list. This one bit me: an annotated tag publishes both a tag object and the commit it peels to, under different hashes. Without refs/tags/<ref>^{} in the pattern list, the remote read returns the tag object while the local read returns the commit; they can never agree, so every check reports fake movement and re-broadcasts forever. test_an_annotated_tag_is_listed_through_the_commit_the_local_read_resolves pins it. All of a repository's tracked refs resolve in a single listing, which test_one_listing_covers_every_ref_rather_than_one_per_ref pins by recording the calls.
  • build_check_refs_model() in factory.py — which branches become tracked refs. The global branch is excluded (it names no branch a worker could check out); so are branches where the repository isn't active.

Mechanical, skim these: docs/docs/reference/configuration.mdx and both docker-compose.yml files are regenerated. dev/specs/.../tasks.md records what landed against the plan, including where the implementation deviated and why.

Alternatives considered: the per-repository body could have lived in tasks.py as the task list originally said. It's a separate package instead, following git/sync.py's precedent of injectable git components consumed by flows — the flows stay thin composition roots.

How to test

# Unit: ref validation, ls-remote parsing, tag handling against a real local repo
uv run pytest backend/tests/unit/git/test_refs_check.py

# Component: claim, lock scope, backoff, broadcast pinning, failure isolation
uv run pytest backend/tests/component/git/test_check_refs.py

# Integration against Gogs: branch advance, force-push, annotated tag move, tag delete
uv run pytest backend/tests/integration/git/test_readonly_refs_check.py

To watch it run for real: create a read-only repository, advance its tracked branch on the remote, and within the configured interval the worker logs Tracked ref moved upstream with the previous and new heads. Confirm the repository's commit in the UI has not changed.

Impact & rollout

  • Backward compatibility: no breaking changes. The new setting has a default and needs no action.
  • Performance: one ls-remote per read-only repository per interval; no content transfer unless a ref actually moved. Five repositories are contacted concurrently. The cycle flow itself runs every minute but the due key means roughly a fifteenth of repositories are looked at on any tick.
  • Config/env changes: INFRAHUB_GIT_READ_ONLY_REFS_CHECK_INTERVAL_MINS (default 15). Present in both compose files and the configuration reference.
  • Deployment notes: safe to deploy. The check starts on its own; there is no migration and no coordinated release.

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change)
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content