test: replace verdaccio and the mock npm servers with an in-process spec-compliant registry

Closed
RO
23d ago
+11667-14895

Replaces verdaccio and the two hand-rolled mock registries the install tests run against with test/npm-registry/, an in-process npm registry built on Bun.serve({ routes }).

The case for this is structural, not a stopwatch win. Verdaccio is a real registry forked as a child process: a boot per test file, a shared mutable on-disk package store that forces every test in a file to run serially and every publish test to rm -rf after itself, and enough startup flakiness that config-version.test.ts spent up to 30 s polling whether it had come up and silently skipped two of its three tests when it hadn't. An in-process registry starts in a socket bind, never touches disk, and every test can have its own, so tests in a file can run concurrently and nothing has state to clean up. Measured on CI (debian 13 x64 release, expected-durations.json vs build 70555), the 11 ex-verdaccio files I captured run in 8.9 s instead of 19.2 s; across the whole test/cli/install/ directory the sum goes from 147.8 s to 141.4 s because the long-pole files were never verdaccio-bound.

The library

test/npm-registry/README.md is the tour. The short version: describe a package's package.json in code and the registry packs the tarball, hashes it, derives the packument, and answers real content negotiation.

await using registry = await new NpmRegistry().start();
registry.define("say-hi", {
  "2.0.0": {
    bin: { "say-hi": "cli.js" },
    dependencies: { "left-pad": "^1" },
    tarball: { "cli.js": "#!/usr/bin/env node\nconsole.log('hi');\n" },
  },
});

What it implements, because the tests needed it:

  • Full and abbreviated ("corgi") packuments with real Accept negotiation, including the derived hasInstallScript.
  • The manifest normalization npm publish applies (bin canonicalization, the implicit node-gyp rebuild install script for a binding.gyp).
  • A deterministic .tgz writer whose tar payload is byte-for-byte npm pack's (node-tar portable: true header encoding, npm-packlist's entry sort, gzip OS byte normalized to 255). tar.test.ts asserts gunzip(buildTarball(src)) equals gunzip(npm pack src) against five committed golden tarballs (plain, bin at 0755, binding.gyp, scoped, and a 117-byte path that crosses ustar's name/prefix split), and pins a known-answer sha512 so a zlib or compression-default change trips there instead of as an opaque integrity diff elsewhere.
  • npm's real tarball URL shape (@scope/name/-/name-1.0.0.tgz), and scoped packuments at both /@scope%2fname and /@scope/name.
  • ETag / If-None-Match / 304, plus Vary: Accept. Cache-Control is sent when configured, but bun does not read that header; its warm-manifest gate is an on-disk cache age.
  • Auth: PUT /-/user/org.couchdb.user:<n>, bearer and basic Authorization, /-/whoami, per-package-glob access rules.
  • Publish end to end: strict Content-Type: application/json (the verdaccio gate the two comments in publish_command.rs cite), attachment decoding, integrity verification of the uploaded bytes, a 403 on republish, deprecate. A failed publish is transactional.
  • npm's 2FA web-authentication flow: the real OTP 401 challenge (www-authenticate: OTP, the exact error string clients match, authUrl/doneUrl) and the doneUrl poll that hands a one-time password back. This replaces a third hand-rolled mock that lived inside bun-publish.test.ts.
  • POST /-/npm/v1/security/advisories/bulk for bun audit, with request Content-Encoding: gzip decoded (which bun audit always sends).
  • Request recording (registry.paths, registry.requests[n].headers) and interception (registry.simulateFailures({ timesPerUrl: 4, status: 500 })), replacing the urls: string[] out-parameter and the expect() calls the old mocks made inside their server handlers.

The class is decoupled from the test harness (it imports only Bun and node builtins); test/harness.ts exports TestRegistry, a thin subclass that preloads the shared fixture tree and owns the bun-specific conveniences (writing a bunfig.toml, the per-test project directory).

The fixtures

test/cli/install/registry/packages/ held 210 packages as verdaccio storage: a hand-written packument JSON each (some hundreds of lines of _attachments, _rev, and integrity strings) plus 287 prebuilt .tgz files. Adding one meant running tar and sha512sum yourself; create-native-binlink-packages.ts was 155 lines of that boilerplate for two trivial packages.

  • All 210 packument JSONs are deleted. The registry derives the packument. I diffed the derived document field by field against every hand-written one before deleting them; the only semantic deltas were things verdaccio had applied at publish time that the registry now applies too (the bin/gyp normalization above) plus four packages whose dist-tags are not derivable, which get a three-line _registry.json. (That diff compared packument fields, including dist.integrity which at that point still came from hashing the original npm bytes; it is not a claim that buildTarball reproduces the deleted tarballs. The npm pack golden vectors are what establish that separately.)
  • 254 of the 287 .tgz files are deleted, exploded into plain source directories (pkg/1.0.0/package.json) the registry packs on demand. The 33 that remain are the ones whose exact bytes are pinned by an artifact that cannot be regenerated (checked-in pnpm-lock.yaml / package-lock.json / yarn migration fixtures, an integrity inlined in a test, a committed snapshot) plus one whose deliberately non-standard internal layout is the thing its test exercises. Each .tgz I kept has a reason.
  • New fixtures never need a .tgz or a packument. A .gitattributes rule marks the tree -text so no platform's EOL conversion can change a packed tarball's bytes, and a single trailing !/test/cli/install/registry/packages/** in .gitignore keeps the whole tree out of the global ignore rules.

Validation

  • The library has its own suite (test/npm-registry/*.test.ts): the golden vectors, the pinned sha512, and the registry's HTTP contract.
  • All 16 test files that used VerdaccioRegistry are migrated and green against a debug build: ~320 tests, zero failures attributable to the registry. (The handful of local reds are environmental and reproducible with no registry configured at all: tests that clone a real GitHub repo through this container's TLS-intercepting proxy, and a cpuTime < 750ms assertion for which a bare bun install here measures 961 ms.)

Three things it surfaced:

  1. Nothing had ever tested bun against a real abbreviated packument. bun asks every registry for application/vnd.npm.install-v1+json; registry.npmjs.org honors that (no scripts, a derived hasInstallScript, a strict field subset) but verdaccio ignores the Accept header and serves the full document. Every install test now runs against the document shape the real registry sends. All of them pass, which is reassuring rather than alarming, but it was an accident before.
  2. bun pm whoami with an invalid token had only ever been tested against verdaccio's non-standard answer (a 200 with an empty body). registry.npmjs.org sends a 401. The registry now 401s, the test asserts bun's 401 output, and a second test reproduces the 200-with-no-username shape with registry.intercept(), so both of bun's code paths for it are now covered deliberately.
  3. Three bun publish tests republished the same name@version within one file and only passed because an rm -rf into verdaccio's on-disk store unpublished it in between. A registry correctly rejects the second publish. Those three now call registry.remove(), which is what that rm really was; the other eighteen rms against registry.packagesPath were dead.

The other half: dummy.registry.ts

All 17 files that used dummy.registry.ts or simple-dummy-registry.ts are migrated too, and both modules are deleted, along with 56 of the 63 flat test/cli/install/*.tgz files they served (the seven that remain are each read by a test of bun's tarball-file dependency protocol, "pkg": "./x.tgz", which has nothing to do with a registry). Each of those tests now gets a fresh in-process registry in its beforeEach and declares its packages in code, so the files that used to share one mutable module-level handler can run their tests concurrently.

Two details from that migration worth a reviewer's attention:

  • The old mock's "fail each URL N times before succeeding" counter actually failed N+1 times (a set-then-check ordering quirk), and the retry test's request-count assertions encode that. registry.simulateFailures({ timesPerUrl }) fails exactly N, so the argument changed by one and the twelve recorded requests stayed twelve.
  • The old mock served every tarball at <root>/<name>-<v>.tgz; a spec-compliant registry serves <root>/<name>/-/<name>-<v>.tgz. Every URL assertion changed shape, which is also the entire diff of the two regenerated security-scanner snapshot files (1,856 lines that sort | uniq -c reduces to four substitutions).

The matrix

test/cli/install/registry-resolver-matrix.test.ts runs five resolver scenarios that have historically broken in exactly one configuration (a transitive peer chain, an optional peer reached from two aliases of one package, a workspace catalog, platform-filtered optional dependencies, and a pair of npm: aliases that point at each other) across {hoisted, isolated} × {cold, warm-on-disk}. The second axis varies BUN_INSTALL_CACHE_DIR (same dir for both passes vs a fresh one for pass 2), which is bun's actual warm-manifest gate; the test asserts the axis is real (a warm pass 2 issues strictly fewer packument requests than a cold one). Every cell of a scenario must produce a byte-identical bun.lock, so resolution can never silently depend on the linker or on whether a manifest came from the network or the warm cache.

Rebase onto 332f7444f9

The branch is squashed and rebased onto main at 332f7444f9. Ten test/cli/install/ files had changed on both sides; here is what each resolution did:

  • #33622 ("make some tests faster") reordered bun-install-lifecycle-scripts.test.ts and rewrote bun-workspaces.test.ts around a semaphore-gated setupTest(). Resolution: take main's version of both files whole, then re-apply only this PR's verdaccio to registry / VerdaccioRegistry to TestRegistry substitution. Jarred's reordering and concurrency machinery are kept exactly.
  • Nine new tests landed on main written against the infrastructure this PR deletes; they are migrated in place with no change in what they assert:
    • bun-install-retry.test.ts: the three redirect-retry tests from #33242. setHandler(h) becomes registry.intercept(h); the handlers, the recorded-URL assertions, and the bar-0.0.2.tgz bytes are unchanged.
    • bun-install.test.ts: the four manifest-tarball-URL-validation tests from #33072. setContextHandler(ctx, h) becomes ctx.registry.intercept(h). (#33159's two transitive-file: tests do not use a registry and auto-merged.)
    • bun-lockb.test.ts and isolated-install.test.ts: the new binary-lockfile-flag test from #33072 and the eight peer-stability/global-store/nested-alias tests from #32182, #33072 and #33646. All already used registry.createTestDir(), which TestRegistry implements, so they auto-merged and pass unmodified.
  • bun-install-tarball-integrity.test.ts: #33245's extra <r> assertions and #33072's three multi-hash dist.integrity tests auto-merged (the new describe brings its own Bun.serve and does not touch the registry); the resolution is a one-line gzipSync import my side had removed.
  • bun-install-security-provider.test.ts: took main's in-memory large-payload path from #33622 (no more temp dir of 3000 .tgz files) on top of this PR's per-test registry.
  • bun-install-registry.test.ts (100 to 10 iterations), bun-lock.test.ts (await using file), .gitattributes, .gitignore, test/tsconfig.json: auto-merged, both sides kept.
  • simple-dummy-registry.ts: deletion kept; NpmRegistry already has [Symbol.asyncDispose].

All nine migrated tests and the two reorganized files pass against a debug build after the rebase. The only local failures are the usual environmental ones the PR body already names (tests that clone a real GitHub/Bitbucket/GitLab repo, and node -p/node-gyp tests that empty PATH and rely on the host's node), which are unchanged by this PR and pass in CI.

Related

no test proof · iteration 39 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/install/bun-add.test.ts test/cli/install/bun-install-lifecycle-scripts.test.ts test/cli/install/bun-install-native-binlink.test.ts test/cli/install/bun-install-registry.test.ts test/cli/install/bun-install.test.ts test/cli/install/bun-link.test.ts test/cli/install/bun-lock.test.ts te…