#44055 test: replace verdaccio with an npm registry on Bun.serve

What I ran

After the rebase. The head is 9e9e3948ec on main c6da4a4d30. Linux x64, 12 vCPU. bun is the canary build e555359cc, which is 8 commits behind that main commit.

• The two tests that only this PR has are the difference between 2516 and 2518: the whoami test and the test of the publish header.
• The runs are after a clean install of test/node_modules from the lockfile of each tree.
• The canary build passes the bytecode test of main on the tree of main. So its bytecode is the bytecode of main.
• I did not run Windows or macOS after the rebase. The Windows machine was not available. CI ran both.
• Build 124229 has one test of this PR that passed on a retry: see the flaky tests below.

After the merge of main 110efa0bc7 (d947d3891c) and the migration of yarn-berry.test.ts (70254020284). Linux x64. bun is bun-linux-x64 of main build 124948, which is 110efa0bc7:

• None of the inputs of the bytecode snapshot changed between c6da4a4d30 and 110efa0bc7, so its values stay.
• The other 31 files were not run again after the merge. CI runs them.

Before the rebase, on the head a4367e507e (then 3b5d5ee45e):

• CI build https://buildkite.com/bun/bun/builds/121299 passed 181 of 181 jobs.
• Windows x64, canary a4f142914: test/packages/registry/ had 183 pass. bun-publish, config-precedence, pnpm-lock-v9 and bun-update-lockfile-sync had 258 pass, 0 fail, 1 skip, three commits earlier.
• A local debug build with ASAN: test/packages/registry/, hoist, npmrc, config-precedence, bun-publish and bun-lockb had 319 pass, 0 fail, when the package had 165 tests.
• With the leftovers of verdaccio in the fixture directory, the 31 files had 2499 pass and 1 fail. The failure was npm manifest cache entries are only reused for the package name they were saved for, which is flaky on main too.

The rebase

• The base moved from a4f1429148 to c6da4a4d30, 88 commits. The merge commit is gone. The 12 commits were replayed, so each has a new id.
• 7 files had a conflict. In 5 of them #44614 changed a line that this PR deletes or renames. test/package.json is the file of main without the two verdaccio lines.
• test/bun.lock is the lockfile of main after bun install --lockfile-only without verdaccio. Its diff has the same 578 lines as before the rebase.
• After the replay, 42 of the 59 files had the same bytes as before the rebase. For the other 17, the patch of this PR had the same lines, apart from lines of main that it deletes and the snapshot.
• The snapshot of libraries.js has new values, because main has a new WebKit: see the bytecode snapshot below.
• Two commits are new. 2d236bfbab: main changed bun-types and test/tsconfig.json (#44614). With them, four lines of the package had a type error. request.bytes() and request.clone() have the types of undici there, and BodyInit is Bun.BodyInit.
• 9e9e3948ec: see "A test that depended on the order of two packuments".

This PR and #33115

#33115 replaces verdaccio too. I did not find it before I opened this PR: my search for open pull requests stopped at 20 results. Both PRs change the same test files, so only one can land as it is. The author of #33115 compared them in https://github.com/oven-sh/bun/pull/33115#issuecomment-5849444725.

A maintainer has to choose. If this PR lands, #33115 can be closed, or reduced to the parts that only it has.

Rulings of the review of #33115 that this PR follows

• A publish is refused with 415 unless the Content-Type is application/json, character for character. bun-publish.test.ts asserts the header that bun sends. The two comments in src/runtime/cli/publish_command.rs stay as they are.
• An error that a handler throws fails the test: stop() throws it.
• The bulk advisory endpoint reads a gzip body, and a test sends one.
• createTestDir() does not remove users or tokens.
• hasInstallScript is true for a script that is not empty.
• time.modified moves on each write.
• A field of an advisory that is passed as undefined keeps its default. The test compares with literal values.
• Nothing here says that bun reads Cache-Control. The registry sends the header because npm does.

What only #33115 has

• The replacement of dummy.registry.ts and simple-dummy-registry.ts. 18 test files import them.
• registry.define(): a package that a test declares in code, with a tarball writer and its golden files.
• The conversion of the fixtures to source directories.
• registry-resolver-matrix.test.ts.

Each of these can go on top of this registry as a PR of its own. So can the three servers in bun-audit.test.ts, config-precedence.test.ts and bun-install-lifecycle-scripts.test.ts that forward to the registry to see or to change a request: intercept and recordRequests do that now.

How each behaviour was established

Compared with registry.npmjs.org, GET and HEAD only

I sent 63 requests to registry.npmjs.org and to this registry with copies of is-number and @types/is-number, and compared the status, nine headers and the body. 49 agree. The requests cover:

• The packument in both forms, HEAD, Range, ?write=true, and a scoped name as @s%2fn, @s/n and %40s%2fn.
• A version by number and by tag, and one that is not there: 404 with the JSON string "version not found: 9.9.9".
• A tarball: GET, HEAD, Range with 206 and 416, If-None-Match, and the 404 for the scope in the file name.
• dist-tags, collaborators, visibility, ping, whoami, the account endpoints, /-/v1/done.
• The same endpoints with a token that npm does not know.

Separate probes gave:

• The abbreviated version is an allow list without libc, description, scripts and time. hasInstallScript is there only when true. The census covered 5763 versions of 9 packages.
• Accept: the registry looks for the media type anywhere in the header, in exact case. It does not weigh q.
• The entity tag is the hex md5 of the body. It is weak on an encoded answer. If-Modified-Since has no effect.
• Encoding: brotli when accepted, else gzip. Bodies of 21 and 26 bytes came back as they are, a body of 63 bytes came back encoded. The registry here starts at 48 bytes.
• Last-Modified is modified rounded up to the second. I checked one packument.
• You must be logged in to publish packages. is the text of npm for a token that it does not know, on collaborators.

The 14 requests that differ

From the npm/registry docs and the npm CLI source, not sent to the live registry

• Login with PUT /-/user/org.couchdb.user:<name>: 201 and {ok, id, rev, token}, or 401 and {"ok":false}.
• Tokens and profile under /-/npm/v1/.
• The publish document, www-authenticate: OTP, the web flow with authUrl and doneUrl, 202 with retry-after.
• The one-time password message. bun compares against this exact text.
• The request sequences of deprecate, unpublish, owner and dist-tag.
• The bulk advisory request and answer.

My wording, because I do not know the text of the registry

• Each 400 Bad Request: ... for a publish body that fails a check, the 415, and the 403 for a read-only token.
• 409 Document update conflict. is the text of CouchDB.
• Publish answers 200 {"ok":true,"id","rev"} and a dist-tag write answers {"ok":"dist-tags updated"}. I recall both from npm output. I did not verify them.
• A restricted package answers 404 to a client that cannot read it. I recall this from npm. I could not probe it, because I know no private package.
• You cannot publish over the previously published versions, You do not have permission to publish and cannot be republished until 24 hours have passed are the texts of npm as I recall them. I did not probe them.
• A publish with a token that the registry does not know answers 401 without www-authenticate. npm sends www-authenticate: Basic, Bearer on collaborators. I do not know what it sends on a publish.

Where the registry is more permissive than npm

• Every user can publish into every scope. npm needs a user or an org with the name of the scope.
• A restricted package needs no paid plan. npm answers 402.
• A package of the storage directory has no maintainers, so every user can add a version to it.

Not implemented: signatures, provenance, OIDC, orgs and teams.

What the tests saw change

Two tests of the 31 files depended on verdaccio. whoami > invalid token asserted its answer, 200 {}. It asserts 401 Unauthorized now, and a new test keeps the 200 {} case with a mock server. ranged peer dependency resolution is stable across installs from bun.lock depended on its speed: see the section on the order of two packuments. Every other test passes unchanged.

bun-publish.test.ts:

• Each rm of a package directory is a registry.packages.delete() of that name. The next section has the reason.
• --ignore-scripts published publish-pkg-4 and removed publish-pkg-5. It passed only because the test before it removed publish-pkg-4. It removes the right package now.
• republishing normally fails matched /403|409|already exists|already present|cannot publish/. It asserts the status and the message.
• A new test asserts that the publish request has content-type: application/json.

harness.ts does not export VerdaccioRegistry. REVIEW.md says to delete code in the PR that makes it dead. The eight files that had a variable verdaccio keep that name, so their diff stays small.

A bug of the registry that the review found: bun publish of a version with build metadata got a 400. bun sends the key 1.0.0 and a manifest with 1.0.0+build.5. The registry accepts that now, and cli.test.ts publishes one with bun.

A checkout that ran the tests on verdaccio

Verdaccio wrote each package that a test published into test/cli/install/registry/packages. That is the fixture directory too, and ignore rules hide what verdaccio wrote. One run of the 31 files on main left 18 packages and .verdaccio-db.json there.

The new registry reads that directory. So it served those packages, and a publish of the same version got 403 You cannot publish over the previously published versions. With the leftovers of that run in place, 15 of the 47 tests of bun-publish.test.ts failed.

• bun-publish.test.ts calls registry.packages.delete() in the 26 places where main removes a package directory. With the same leftovers in place the file passes.
• The ignore rules stay as they are on main, so the leftovers stay out of git status.
• The builds of this PR did not show this: a checkout of CI has no leftovers.

Layout

The package has the layout of test/packages/s3-server, which came to main while this PR was open: the sources in src/, the tests in test/, and the helpers of the tests in test/helpers.ts. index.ts does not import the harness. test-registry.ts has the class for the tests of bun, and the alias "registry" points to it.

One port, one registry

Bun.serve sets SO_REUSEPORT when it gets development: false (src/runtime/server/ServerConfig.rs:716). The registry passed that option, so a second registry on the same explicit port started with no error. It passes reusePort: false now, and the second one fails with EADDRINUSE. The new test fails without the option.

The tests listen on port 0. On Linux, 20000 listeners on port 0 with SO_REUSEPORT got 20000 distinct ports, so I have no sign that a test shared a port. I did not check that on macOS or Windows.

The bytecode snapshot

bundler_bytecode_portable.test.ts pins the bytecode of a bundle of 64 libraries from test/node_modules. The bundle holds each package that a library can load, also one that it loads in a try block and does not declare.

test/ uses the isolated linker. A package that is not declared is still found through node_modules/.bun/node_modules, which has one version of each installed package. Two libraries of the bundle load a package that way:

So 7 modules of 3 packages leave the bundle: aws4@1.13.0, debug@4.3.4 and ms@2.1.2. Nothing is added. To check this, the lockfile of c6da4a4d30 and the lockfile of this PR were installed in two directories, and libraries.js was built in each with the command of the test. The sha256 of the two .js outputs are the old and the new value of the snapshot. A metafile names the two importers above.

The bytecode is 32632 bytes smaller: 21179744, where main has 21212376. Before the rebase the two numbers were 21174272 and 21206912, with the WebKit of that time. The new values are the same on every platform, which is the case where the comment at the top of that file says to update the snapshot.

Test timings

"Before" is main at c6da4a4d30, where the 31 files fork verdaccio. "After" is this PR on the same commit.

Local

• Linux x64 VM, 12 vCPU (Intel Xeon, 2.90 GHz), 16 GB. bun is the canary build e555359cc for both sides.
• Each file ran alone: bun test --timeout=90000 <file>. The two trees took turns on each file. The order of the two changed with each file and each round. 5 rounds, 310 file runs.
• Wall-clock is from the start of bun test to its exit. CPU is the user and system time of all processes of the VM in that time, from the cgroup. It has the verdaccio process, which the test does not wait for.
• The table has the median of the runs that exited with 0, and the lowest and the highest in brackets. Seconds.
• The after side is the commit 2d236bfbab. The commit after it changes one test of isolated-install.test.ts and adds no wait for time.
• The VM lost CPU time to other VMs in each round (steal time), so the wall-clock has noise. The rounds agree: before is 1.7 to 2.1 times after in each one.

Each round, the sum over the files where both sides exited with 0:

Runs that are not in the tables:

• 6 of the 155 runs with verdaccio did not start it. The log has Verdaccio exited with code 2 and signal null. Code 2 is the exit of verdaccio when it cannot listen (build/lib/bootstrap.js). start() waits for a message that does not come. Five runs ended at the hook timeout of 90 s. bun-install-registry.test.ts waits at module scope, where no timeout applies: I stopped it after 678 s.
• I started verdaccio on a port that was in use, to check: the same line, and start() did not settle.
• 1 of the 155 runs of this PR had one failing test: see the next section.

CI

Build 124221 is main at c6da4a4d30. Build 124229 is this PR on the same commit. Builds 124209 and 124201 are main, one and four commits earlier, with the same 31 files. Each number is the sum over the 31 files of the duration that the runner logs for a file. I read the logs with the parser of scripts/update-test-durations.mjs. Seconds.

• The 222.6 s of build 124221 have one run of 90 s: isolated-install.test.ts on shard 8. The log has Verdaccio exited with code 2 and signal null and the hook timeout. The second attempt passed.
• The main builds have no test jobs for ASAN and macOS.
• A build is one sample. Apart from the 90 s run, the three main builds differ by up to 47 s on a lane (alpine).
• Not each file is faster in CI. On each lane, 1 to 6 of the 31 files took longer than in each of the three main builds. The largest differences are on alpine: bun-install-registry.test.ts took 37.3 s against 27.9 to 29.2 s, and bun-update.test.ts took 7.2 s against 2.5 to 2.7 s. That is one sample, and I do not know the cause. Locally each file is faster.
• The four test files of test/packages/registry/ take 1.5 to 4.8 s on a lane.

Import cost, CPU, the lowest of the runs. Measured before the rebase, on a build of 367d939d9:

A test that depended on the order of two packuments

isolated-install.test.ts, ranged peer dependency resolution is stable across installs from bun.lock. The root depends on normal-dep-and-dev-dep (which has no-deps@1.0.1), on two-range-deps (no-deps@^1.0.0) and on peer-deps-fixed (peer no-deps@^1.0.0). The test needs 1.0.1 and 1.1.0 in the graph.

• bun resolves ^1.0.0 to 1.1.0 when the packument of two-range-deps is there first. When the packument of normal-dep-and-dev-dep is there first, 1.0.1 is in the graph and ^1.0.0 resolves to it.
• I delayed one packument by 40 ms, with 12 installs for each case. A slow two-range-deps gave one version 12 times. A slow normal-dep-and-dev-dep gave both versions 12 times. No delay gave both 7 times and one 5 times.
• On main with verdaccio the test passed 50 of 50 runs. With the new registry it failed 54 of 100 runs alone, and 10 of 20 runs of the file. It fails with the older bun build 367d939d9 too (15 of 40), so the rebase did not bring it.
• CI did not show it, before and after the rebase. CI sets BUN_INSTALL_CACHE_DIR, so the tests of a file share one manifest cache. With that variable the file passed 10 of 10 local runs. Without it each test has its own cache.
• My local runs before the rebase did not show it. They were on a host with a load average of 400 to 900.
• The test has its own registry now. Its intercept answers for normal-dep-and-dev-dep after bun asked for no-deps, which bun does when it has the packument of two-range-deps. There is no wait for time. 100 of 100 runs alone and 20 of 20 runs of the file pass.
• After that change: 5 passes over the 31 files, and 5 runs in a row of each file. 310 file runs, 0 failures.

Other tests can depend on the order in the same way. hoisting > peers > it should hoist 1.0.1 when peer * does: see the flaky tests below. The cause is in bun install, not in the registry: one package.json gives two different trees.

Other things I found

• Before the rebase, test/bun.lock on main was stale: bun install --lockfile-only on an unchanged main dropped five nested react entries, and I kept them by hand. The lockfile of main is current now. The lockfile diff here is what bun writes: verdaccio and its dependencies go, and debug hoists to 4.4.0.
• manifest conditional requests > a changed etag returns 200 and the new etag is cached failed once on unchanged main and passed on the next run. It uses its own mock server, not the registry. It is in test/flaky-tests.txt.
• On my local debug build, Bun.serve adds content-type: application/octet-stream to an answer that has no body: a 304, and a 401. Two release builds, one of 367d939d9 and one of 36cd1514ec, do not. I did not find the cause and I saw it on one local build only. The tests here assert the exact headers on the Response of the registry, and on the wire only the headers that the registry sets.
• Bun.serve sets SO_REUSEPORT when it gets development: false. The type of reusePort says that the default is false.
• bun install gives another tree for the same package.json when the packuments arrive in another order: see the section on the order of two packuments.
• bun run and bun test use a whole core, and do not end, while a top-level await waits for a promise that nothing settles. A file with await new Promise(() => {}) used 9.9 s of CPU in 10 s. node prints Warning: Detected unsettled top-level await and exits with 13. The loop is wait_for_promise in src/jsc/event_loop.rs. This is how bun-install-registry.test.ts on main hangs when verdaccio does not start.

Flaky tests in files that this PR edits

Two tests of migrated files passed on a retry in the earlier builds of this PR. Main has both without this change. In build 121299, the last one before the rebase, no file of this PR was in the flaky annotations. Files outside this PR were.

• bun-install-registry.test.ts, hoisting > peers > it should hoist 1.0.1 when peer *, on Windows 11 aarch64, with a-dep@1.0.9. The same test fails the same way on the same lane in the main builds https://buildkite.com/bun/bun/builds/120426 and https://buildkite.com/bun/bun/builds/120659. I could read 11 main builds. After the rebase it passed on a retry on ubuntu 25.04 aarch64 in build 124229, again with a-dep@1.0.9. The version that it gets depends on the order of the packuments.
• bun-lock.test.ts, peer no published version satisfies > declared by a registry package. It runs against its own mock server. test/flaky-tests.txt lists it at 34 of 80 builds.

I cannot say that the rate of the first one is unchanged. On my machine it failed 1 time in 150 runs with this registry, and 0 times in 29 runs with verdaccio.

A third test failed in my runs, and not in the builds of this PR:

• bun-install-registry.test.ts, npm manifest cache entries are only reused for the package name they were saved for: failed to open manifest file ... ENOENT. It failed in 1 of 8 runs of the file. Alone, it failed in 2 of 40 runs with this registry, and in 5 of 40 runs on main with verdaccio. bun install removes a manifest file that is not valid and saves the new one in a task of the thread pool (Serializer::save_async in src/install/npm.rs). I found nothing that waits for that task before the exit. The test reads the file after the exit. test/flaky-tests.txt has failed to open ... for this file.

Reviews of the diff

My review of the diff raised 9 concerns. 8 were addressed in full: the header of a publish, an error in a handler, the export of the old name, pack(), the claims about npm, #33115, the flaky tests, and the time of the tests in cli.test.ts. One is addressed in part: the review named six follow-ups, and these Notes list the three that I can describe.

The reviews on GitHub raised 12 more. All are addressed:

• The URL of a registry names the host that it listens on. It was localhost for each host.
• A checkout with the leftovers of verdaccio failed the publish tests: see the section above.
• The ignore rules for the leftovers stay.
• intercept gets a copy of the request, so it can read the body of a request that the registry answers.
• The web login test fails with the output of bun when bun exits before it asks for the session. Before, it waited for the timeout.
• Each test of the package disposes its registry when an assertion fails.
• VerdaccioRegistry is removed. My review asked to keep the export for open PRs. The review on GitHub asked to remove it, and REVIEW.md says the same.
• A request that the registry refused left a part of its change in the package: a PUT to /<name>/-rev/<rev> with a tag that is not valid, and a deprecation with a message that is not a string. The checks come before the first change now.
• In a search, a qualifier that matched took away the miss of a word before it. A package is found when each word matches.
• A storage that is not a directory gave a registry that answered 404 for every package. The constructor throws now, and cli.ts prints the message. A read that failed does not stay as the state of the package.
• cli.ts ended with a stack trace for a --user that the registry refuses, for an option that it does not know, and for a port that is in use. One handler prints the message and the usage for each refusal now.
• cli.ts took an empty --port as port 0, and it took 0x50 and 1e3. The value must be digits now.

One more finding got no change: a scope directory that another process replaces with a file while the registry reads it. Nothing writes into the storage while a registry runs, and a check before the read has the same window. The reviewer withdrew the finding.

Related open PRs

These change VerdaccioRegistry in test/harness.ts. This PR removes that class, so git reports a conflict, and the problem each one addresses is not there with the new registry:

• #36352: stop() does not end the process. The new stop() closes the server.
• #38891: the random port and a verdaccio that exits early. The new registry listens on port 0 in the process of the test.
• #40219: createTestDir() removes the users. The new registry keeps them.
• #40766: run verdaccio under node in CI.

These edit the line of hoist.test.ts that imports VerdaccioRegistry. Git reports a conflict on that line. The new line imports TestRegistry from "registry":

• #43878
• #38976

I looked at the 500 open PRs that changed last. 72 of them change a file under test/cli or test/regression, the harness, or a test with install, publish or registry in its path. I read the diff of each. No PR other than the ones above adds or changes a line with VerdaccioRegistry.