fix(crawl): surface API errors and failed pages from crawl status

RA
3d ago
+217-7

Summary

firecrawl_check_crawl_status, and the polling inside firecrawl_crawl, read job state through the SDK's HTTP layer directly. That path has two defects:

  • A non-2xx answer reaches the caller as the raw transport error. Checking an unknown or expired crawl ID returns Tool 'firecrawl_check_crawl_status' execution failed: Request failed with status code 404 instead of the API's Job not found.
  • While following next links, a page that answers success: false stops the loop without an error. The call returns the pages read so far with status: "completed" and next: null, so a partial crawl looks finished.

Changes

  • Add getCoreJson, which runs a GET through the SDK's HTTP layer and raises a non-2xx answer as CoreHttpError carrying the API's error message, HTTP status and agent hints. Because the status is preserved, the existing 401 credential recovery still applies.
  • Use it for the first status request and for every pagination request.
  • Throw when the first response has success: false, and when a later page has success: false, instead of breaking out of the loop.
  • Add tests/crawl-status.test.mjs with a fake API covering a two-page crawl, an unknown job, and a page that fails mid-pagination.
  • Add a changelog entry.

Test plan

pnpm run build
node --test tests/crawl-status.test.mjs

Before the change: 3 tests, 1 pass, 2 fail (reports the API error for an unknown crawl, fails when a result page cannot be read). After the change: 3 tests, 3 pass. npx eslint src/index.ts and npx tsc --noEmit report no problems.

The two-page test passes both before and after the change, so it guards against regressions in the join behavior.

Summary by cubic

Makes crawl status checks surface the API's real error messages and fail loudly when a paginated result can't be read.

Previously, firecrawl_check_crawl_status (and polling inside firecrawl_crawl) returned raw transport errors like Request failed with status code 404 for unknown jobs, and silently returned partial results as a completed crawl when a follow-up page failed. Now both cases fail with the API's own error message while preserving the HTTP status so the existing 401 credential recovery still applies.

  • Throws CoreHttpError carrying the API's error message, status, and agent hints for non-2xx responses and for 2xx replies with success: false.
  • Raises an error when the initial response or any paginated page returns success: false.
  • Adds tests/crawl-status.test.mjs with a fake API covering a two-page crawl, an unknown job, a mid-pagination failure, and a first-response failure.