fix(host-service): count PRs a merge queue closed as merged

JO
3d ago
+452-3

Summary

  • The host now counts a PR as merged when GitHub shows it closed but its squashed commit is on the base branch. Graphite's merge queue merges this way.
  • With this, the usage and leaderboard counts of merged agent PRs, and the cost per PR, include PRs merged through Graphite.

Why / Context

Graphite's merge queue runs CI on a draft PR, pushes the squashed commit to main itself, then closes the original PR. GitHub never sets merged_at on either PR. The host stored these PRs as closed, and countAgentPrsByDay (which reads pull_requests.merged_at) left them out.

Real example, remilabs/lidar-server#309:

  • GitHub shows it CLOSED, merged: false, closed by graphite-app.
  • Its draft CI PR, #313, is also closed and not merged.
  • main has bcca8af Optionally keep a small roof face… (#309).
  • GET /commits/bcca8af/pulls returns nothing, so GitHub doesn't link the commit to the PR.

How It Works

  • New utils/landed-merge/: for a closed, unmerged PR, lists commits on its base branch from 3 h before the close to 10 min after. A commit whose subject line ends in (#<number>) means the PR landed. The commit's time becomes mergedAt.
  • PR refresh: runs that lookup for closed, unmerged PRs and stores a match as merged. Each PR is looked up once per host process. A row already stored merged is not looked up again, and stays merged when GitHub returns CLOSED (GitHub never turns a merged PR into a closed one).
  • Startup pass: at host start, checks linked PRs from the last 30 days that are stored closed with no mergedAt. This covers rows stored before this change and PRs of archived workspaces, which the refresh skips.
  • github-query now reads closed_at and base.ref into the PR node, as optional fields.
  • Lookups go through fetchFromGitHub (gh first, then Octokit, behind the availability gate). A failed lookup logs a warning and leaves the PR closed.

Manual QA

  • Ran the new lookup with real gh against remilabs/lidar-server:
    • #300, #307, #308, #309 were found on main with their commit times.
    • #313 (Graphite's draft CI PR) was not found, so it isn't counted.
  • Ran the startup pass (backfillLandedMerges) with real gh on a copy of a real host DB:
    • #300, #307, #308, #309 went from closed to merged, dated by their commits.
    • countAgentPrsByDay(db, 30), which feeds the usage and leaderboard counts, went from 0 to 4.
  • Not checked by eye in the desktop usage page; that page reads the same countAgentPrsByDay output.

Testing

  • bun run typecheck (host-service): passes
  • bunx biome check packages/host-service/src/runtime/pull-requests: clean
  • bun test src/runtime/pull-requests src/trpc/router/usage/history: 198 pass
  • New tests:
    • a Graphite-style closed PR becomes merged, with only one commits lookup over two refreshes
    • a revert commit that mentions the PR doesn't count
    • the startup pass marks a PR of an archived workspace as merged
    • subject-line matching (rejects (#3090) and a mention in the message body)
  • Full bun test in host-service: 2884 pass, 1 fail. The failure is workspace PR history > history accumulates across branch moves, an existing timing flake: after an unlink, history is sorted by linkedAt alone, and on a slow full run both links can get the same millisecond. It passes 8/8 run on its own, and the new code doesn't run for the open PRs it uses.

Known Limitations

  • Only commits whose subject ends in (#N) are found (squash merges, which is what Graphite does). A merge whose commit has a different subject won't be counted.
  • One page of 100 commits per lookup. On a base branch with more than 100 commits in the 3 h window, a PR could be missed.
  • These PRs now show as merged instead of closed in the host's own PR views. Views that read GitHub directly still show closed.

Risks / Rollout

  • Risk: low. The change only adds GitHub calls for closed, unmerged PRs (one per PR per host process; the conditional ETag cache makes repeat calls cheap), and only ever turns closed into merged.
  • Rollback: revert this PR. Rows already marked merged stay merged, which is correct for PRs that really landed.

Summary by cubic

Fixes PR counting so PRs closed by a merge queue still count as merged in usage and leaderboard stats, even though GitHub never sets merged_at.

For closed, unmerged PRs, the host now checks the base branch for a commit whose subject ends in (#<PR number>) near the close time and stores the PR as merged with that commit's time. A one-time pass at host start covers closed PRs from the last 30 days, including those of archived workspaces. Each PR is looked up once per host process, and failed lookups leave the PR closed.

  • Lookups only match squash-merge subjects; other merge styles won't be counted.
  • A stored merged PR stays merged when GitHub reports it closed, since GitHub never reverses a merge.

Summary by CodeRabbit

  • Bug Fixes
    • Closed pull requests that were merged through the repository’s merge queue are now recognized as merged, with the landed commit time recorded when available.
    • Existing merged status is preserved when GitHub reports a pull request as closed.
    • Recent closed pull requests linked to archived workspaces are also checked during startup backfill.