fix(connectors): stop hydration after rate limits

Merged
WA
2mo ago
+371-63

Summary

  • stop deferred connector hydration after a structured upstream rate limit so later batches do not amplify one provider throttle into per-document failures
  • preserve the typed provider error and retry delay for the existing connector backoff scheduler while leaving ordinary document failures isolated
  • use GitHub’s documented object media type for file metadata and a byte-bounded raw blob stream for large content, surfacing non-404 failures and oversized blobs explicitly
  • add connector and sync-engine regressions for the exact rate-limit, large-file, absence, permission, and byte-cap behaviors

Testing

  • bun run --cwd apps/sim test -- connectors/github/github.test.ts lib/knowledge/documents/utils.test.ts lib/knowledge/connectors/sync-engine.test.ts
  • bun run --cwd apps/sim type-check
  • bun run agent-stream-docs:generate
  • bun run skills:sync
  • bun run lint
  • bun run apps/sim/scripts/check-block-registry.ts origin/staging
  • bun run check:audits