fix(connectors): stop hydration after rate limits
Merged
WA
2mo ago+371+371+012345678901234567890123456789-63-63-01234567890123456789
Summary
- stop deferred connector hydration after a structured upstream rate limit so later batches do not amplify one provider throttle into per-document failures
- preserve the typed provider error and retry delay for the existing connector backoff scheduler while leaving ordinary document failures isolated
- use GitHub’s documented object media type for file metadata and a byte-bounded raw blob stream for large content, surfacing non-404 failures and oversized blobs explicitly
- add connector and sync-engine regressions for the exact rate-limit, large-file, absence, permission, and byte-cap behaviors
Testing
bun run --cwd apps/sim test -- connectors/github/github.test.ts lib/knowledge/documents/utils.test.ts lib/knowledge/connectors/sync-engine.test.tsbun run --cwd apps/sim type-checkbun run agent-stream-docs:generatebun run skills:syncbun run lintbun run apps/sim/scripts/check-block-registry.ts origin/stagingbun run check:audits