fix(crawl): return pages for a job ID, resume --wait by ID, exit 1 when a waited crawl fails

RA
17h ago
+282-123

Three gaps made a detached crawl unusable from scripts and agents:

  • firecrawl crawl <job-id> printed only {id, status, total, completed, creditsUsed, expiresAt} and dropped the pages, so a crawl started without --wait could never be read back from the CLI.
  • firecrawl crawl <job-id> --wait didn't wait: the status branch ran before the wait logic.
  • crawl --wait exited 0 when the crawl ended failed or cancelled; the status was only visible inside the JSON.

Now:

  • Job ID: crawl <id> (or --status) prints the full job from getCrawlStatus, including data (the pages collected so far, auto-paginated by the SDK) and next. -o and --pretty apply as for a completed crawl.
  • Resume: crawl <id> --wait polls that job until it's completed, failed or cancelled, honouring --poll-interval, --timeout (both must be positive numbers) and --progress. Each poll fetches one status page, and all pages are fetched once the job ends. A timeout prints the command to resume with.
  • Exit status: with --wait, a crawl that ends failed or cancelled still prints the job JSON, then writes Error: Crawl <status>. to stderr and exits 1. Reading a job's status without --wait keeps exit 0 whatever the status, and a successful --cancel exits 0 even with --wait.
  • The unused human-readable status formatter (its guard could never be true) and the CrawlStatusResult type are removed; --help describes the job ID and --wait behaviour.

Show me (local mock API, a job that ends failed with one page)

$ firecrawl crawl 2222…2222
{"id":"2222…2222","status":"failed","completed":1,"total":3,…,"data":[{"markdown":"p1","metadata":{"sourceURL":"https://a.com"}}]}
exit=0

$ firecrawl crawl 2222…2222 --wait
{… same job …}
Error: Crawl failed.
exit=1

$ firecrawl crawl https://a.com --wait
{… same job …}
Error: Crawl failed.
exit=1

Validation

  • pnpm run type-check, pnpm run build and pnpm test pass (836 tests); prettier clean on the changed files.
  • New tests: a job ID returns its pages; --wait on a job ID polls to a terminal status without starting a crawl; a timeout names the resume command; a waited crawl that ends failed or cancelled prints the job and sets exit code 1; reading a failed job without --wait, or cancelling with --wait, leaves the exit code alone; waits poll one page at a time; invalid wait flags are rejected before polling.

Open in Capy

Summary by cubic

Fixes three gaps in the firecrawl crawl CLI that made detached crawls hard to use from scripts and agents.

  • crawl <job-id> now prints the full job from the status endpoint, including collected pages (data) and next, instead of a stripped-down summary.
  • crawl <job-id> --wait polls the existing job to a terminal status instead of returning immediately; a timeout prints the resume command to rerun, and --progress shows live progress while waiting.
  • A waited crawl that ends failed or cancelled now exits 1 and writes Error: Crawl <status>. to stderr, while reading status without --wait, or cancelling with --wait, keeps exit 0.
  • --poll-interval and --timeout are validated as positive numbers before a wait starts.
  • Removed the unused human-readable status formatter and CrawlStatusResult type, and updated --help text to describe the job ID and --wait behavior.