fix: Crossref 429 resilience + salvage sweep + OpenAlex select fix (v1.3.1)

Merged
TO
4mo ago
+320-5

Problem

A parallel batch of reference checks against Crossref's anonymous pool reliably hits 429 storms: 10 concurrent query.bibliographic searches get rate-limited, the fixed 300ms/1200ms retry backoff re-enters the same window, all attempts die, and the majority of a real bibliography lands as check_failed ("could not be checked"). Reproduced with a realistic 35-ref APA list: 14/35 check_failed, 58× HTTP 429.

Separately, every OpenAlex /works lookup silently failed with HTTP 400 because is_oa is not a valid top-level select field — no citation ever got citation-count / OA enrichment.

Fix

  • fetchRetry: honor Retry-After (delta-seconds or HTTP-date, capped 30s) on 429 and share the wait across all requests to the same host via a per-host cooldown map, with jitter so a released batch does not stampede back in lockstep. 5xx/network/timeout keep the existing backoff.
  • Salvage sweep (quickCheck + checkDocument): refs that still ended check_failed are re-checked once, sequentially, after the burst has drained, under a 60s wall-clock budget. Transient failures self-heal instead of surfacing to the user.
  • OpenAlex: select open_access (valid) and read open_access.is_oa.

Result (same 35-ref list, anonymous pool)

Before: 14 check_failed / 15 verified, OpenAlex 100% HTTP 400.
After: 0 check_failed / 25 verified / 9 partial / 1 not_found, OpenAlex enrichment working. 429s still occur but are absorbed.

Tests

10 new tests (http-retry, salvage, openalex select); full suite 138 pass.

Summary by cubic

Improves Crossref rate-limit handling and restores OpenAlex enrichment, reducing false check_failed results during parallel checks. Honors server backoff and retries failed refs once to self-heal transient errors.

  • Bug Fixes
    • 429 resilience: fetchRetry honors Retry-After (seconds or HTTP-date), applies a shared per-host cooldown with jitter, and keeps existing backoff for 5xx/network errors.
    • Salvage sweep: after parallel checks, re-run check_failed refs once, sequentially, under a 60s budget in quickCheck and checkDocument.
    • OpenAlex: select open_access and read open_access.is_oa (replaces invalid is_oa select) to restore citation-count and OA enrichment.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of rate limits by honoring server-provided retry timing and coordinating retries for requests to the same host.
    • Added a follow-up check for references affected by temporary network failures.
    • Fixed OpenAlex enrichment errors by using the correct open-access response field.
  • Tests
    • Added coverage for rate-limit retries, OpenAlex lookups, and recovery from temporary citation-check failures.