fix(scrape): keep every page when multi-URL filenames collide

BR
2mo ago
+170-3

What's wrong

firecrawl scrape with several URLs saves each page to .firecrawl/<host>-<path>.md (urlToFilename). The name drops the query string and turns / into -, so different URLs can map to the same file:

$ firecrawl scrape "https://example.com/list?page=1" "https://example.com/list?page=2" \
    https://example.com/a/b https://example.com/a-b
Scraping 4 URLs...
[1/4] Saved: .firecrawl/example.com-list.md
[2/4] Saved: .firecrawl/example.com-list.md
[3/4] Saved: .firecrawl/example.com-a-b.md
[4/4] Saved: .firecrawl/example.com-a-b.md

Completed: 4/4 succeeded
$ ls .firecrawl
example.com-a-b.md  example.com-list.md

The run reports all four pages as saved, but only two files exist. Whichever request finished last overwrote the other page. Paginated URLs such as ?page=N or ?id=N are a common reason to pass several URLs to one command, and they always hit this.

Fix

Filenames are chosen before the requests start, in the order the URLs were given. If a name is already used in the batch, the later URL gets -2, -3, and so on (example.com-list-2.md). URLs that don't collide keep the same names as before, so existing scripts that read .firecrawl/<host>-<path>.md are unaffected.

Tests

New src/__tests__/commands/multi-scrape-filenames.test.ts (mocked client and fs):

  • the four colliding URLs above are written to four distinct files, each with its own content. This fails on main, where only two files are written.
  • URLs that don't collide keep their plain names.

pnpm run format:check, pnpm run type-check, pnpm run build and pnpm test (642 tests) pass. Under heavy local load, some subprocess tests in alexandria-beta.test.ts timed out on both main and this branch. They pass with --maxWorkers=4.

This PR was prepared with an AI coding agent (Claude). I reproduced the bug and checked the fix and tests before opening it.

Summary by cubic

Fixes firecrawl scrape silently losing pages when URLs bundled in one run collide on their output filename. URLs that differ only in their query string (?page=1 vs ?page=2) or in / versus - (/a/b vs /a-b) mapped to the same .firecrawl/<host>-<path>.md file, and the last request to finish overwrote the rest while the run still reported every URL as saved.

  • Filenames are assigned up front in URL order, with a -2, -3, ... suffix for names already used in the batch. Plain names are reserved first so a suffixed name can't take another URL's plain name, and names are compared case-insensitively to match macOS and Windows.
  • Non-colliding URLs keep their existing names, so scripts that read .firecrawl/<host>-<path>.md are unaffected.
  • Adds tests covering colliding, non-colliding, plain-name-reserved, and case-only-colliding batches.