fix(host-service): stop spawning gh for every repo while GitHub is unreachable
Problem
A user's host-service log showed thousands of getaddrinfo ENOTFOUND api.github.com failures. Each one was a gh spawn that sat on its 10 s timeout, got SIGTERMed, fell back to Octokit, and failed again on the same resolver. The per-repo cache backoff in cachedGitHubFetch can't help here: the failure is a property of the host's network, not of any repo, and with 39 projects polled on a cadence the spawn rate stayed high for as long as DNS was broken. On a machine already slow to exec (endpoint security, VPN), that is a lot of extra load on the process that carries terminal I/O.
Change
- Add one reachability gate across the PR runtime's head lookups, open-PR sweeps, reviews, checks, and merge-queue lookups. Transport failures hold polling for one minute, doubling to a 30-minute cap. Concurrent failures during one hold neither extend it nor produce additional hold logs.
- Recognize Node/undici transport errors, actual gh resolver-failure wording, and killed gh requests. Transport failures skip the Octokit fallback; HTTP failures retain it.
- Explicit refreshes bypass the hold. Successful gh and Octokit lookups reset it. Gate-blocked cache entries are discarded so they can retry immediately after recovery; ordinary failed requests retain their existing cache backoff.
Testing
- Runtime suite: 77 pass, 0 fail. Host-service typecheck, full repository lint, and plugin validation pass.
- Three new recovery regressions were run before and after the final fix: held repo → successful probe → immediate normal retry, plus separate successful Octokit review/check and merge-queue fallbacks resetting the next hold to one minute. All three failed before the fix and pass afterward.
- Earlier real-gh outage harness evidence is in the PR comments: real manager and migrated SQLite, real gh 2.98 with a per-process invalid GitHub hostname. Eight concurrent failures produced one hold; a ninth held request spawned nothing; an explicit probe passed through; the next outage window doubled; HTTP errors still fell back.
- The final recovery tests use simulated providers and clocks. This is host-runtime verification, not a CDP UI test or a CPU profile.
https://claude.ai/code/session_01UWyngsh3WSdEM6kmf1P8s5
Summary by cubic
Prevents the PR runtime from spawning gh for every repo while GitHub is unreachable. A reachability gate now holds all GitHub lookups on transport failures, backing off from 1 to 30 minutes, instead of each repo paying a 10-second gh timeout plus a doomed Octokit fallback.
Bug Fixes
- Transport failures (Node/undici codes,
ghstderr phrasings, exec timeout) trip the gate and skip the Octokit fallback since it would hit the same broken network. gh 2.98's own resolver wording ("error connecting to", "check your internet connection") is now recognized too, so a dead resolver trips the gate on the firstghcall rather than after the fallback. - HTTP failures (404, 403, 5xx) never trip the gate and still fall back to Octokit.
- Concurrent failures of one outage open a single hold and log once; failures during an active hold don't extend it.
- Review/checks/merge-queue lookups in polling now go through the gate too. Explicit refreshes probe through an active hold, so a user-driven action can reopen the gate early.
Summary by CodeRabbit
- Bug Fixes
- Improved pull request lookups during GitHub network outages by temporarily suppressing repeated requests.
- Added automatic retries with progressively longer wait periods after connectivity failures.
- Restored normal lookups promptly after a successful connectivity check or fallback response.
- Preserved fallback behavior for GitHub API responses, including rate limits, missing resources, and other HTTP errors.
- Allowed explicit refreshes to bypass an active outage hold when requested.
- Tests
- Added coverage for outage detection, retry timing, concurrent failures, transport failures, detail lookups, and API fallback behavior.