install: read a registry URL the way new URL() reads it

Closed
RO
25d ago
+217-17

Problem

  • BUN_CONFIG_REGISTRY=http://127.0.0.1:A#@127.0.0.1:B/ bun install sends GET /no-deps to B with Authorization: Basic base64("127.0.0.1:A#"). new URL(registry).host is 127.0.0.1:A. http://A\@B/ behaves the same, from .npmrc, bunfig and --registry too. For the env vars \@ is a 1.4 regression.
  • Cause: NpmRegistry::from_url (src/options_types/schema.rs:158), where every registry URL string is split, uses bun_url::URL::parse, whose credential and host scanning stop only at / and ?. It takes 127.0.0.1:A# for user:password and B for the host.

Fix

  • from_url parses the string with WTF::URL first. When it has a host but no password, the registry is the string as written, as today (.npmrc //host/ keys match these bytes, and Scope::set_url normalizes later). With a password, the registry is scheme://host/path/ from the WTF::URL components, the form href_without_auth produced, and the credentials are its username and password. A string WTF::URL rejects or finds no host in (localhost:4873) goes to URL::parse as before.
  • WTF::URL serializes credentials percent-encoded (p@ss becomes p%40ss), so they are decoded with the new PercentEncoding::decode_lenient_alloc. http://carol:s3%40cret@host/ now sends carol:s3@cret, as curl does.
  • The two same-origin checks in PackageManagerOptions.rs that carry a .npmrc token over to an env or --registry URL now read that URL as set_url does (keeps_origin). With the lenient parser they named the host after #@ while the request went to the one before it, so B's token reached A.
  • Verified: test/cli/install/registry-url-authority.test.ts, 8 of 11 cases fail on 1.4.3 or on the first revision of this change. Also the install suites that use a registry URL (notes). The only failures need the public registry or GitHub.

Background

  • Every registry source reaches from_url. Scope::from_api (src/install/npm.rs) then builds the Authorization header, and Scope::set_url normalizes the href through WTF::URL. The host came from the first step.
  • Bun has two URL parsers. WTF::URL (WebKit) is the WHATWG parser behind new URL(). bun_url::URL::parse slices an href that is already normalized. #39933 made the S3 endpoint take this path.
  • #39603, #39063, #38834 and #40423 are open in the same function. None changes where the host comes from (notes).
Notes

Repro on release 1.4.3 with two Bun.serve recorders on 127.0.0.1 (run without HTTP_PROXY):

BUN_CONFIG_REGISTRY=http://127.0.0.1:A#@127.0.0.1:B/
  A: []
  B: [GET /no-deps  Authorization: Basic MTI3LjAuMC4xOjQ0MzMxIw==]   (base64 "127.0.0.1:A#")
BUN_CONFIG_REGISTRY=http://127.0.0.1:A\@127.0.0.1:B/
  A: []
  B: [GET /no-deps  Authorization: Basic ...XA==]                   (base64 "127.0.0.1:A\")
BUN_CONFIG_REGISTRY=http://127.0.0.1:A?@127.0.0.1:B/
  no request: "manifest URL http://127.0.0.1:A/no-deps is not on registry http://127.0.0.1:A/?@.../"

With the change the #@ form requests GET /no-deps on A with no Authorization. The \@ form requests /@127.0.0.1/no-deps on A: \ is / to WTF::URL, so @127.0.0.1:B/ is a path prefix, and Scope::from_api strips the :B as it strips any :key path suffix. The ?@ form still sends nothing, now for the same reason as before (a registry URL with a query matches no manifest URL). The test pins the first two and the absence of a request for the third.

The same function reads bunfig.toml, .npmrc and --registry; the test covers those with the \@ form.

Why a WTF::URL result without a host falls through: WTF::URL reads localhost:4873 as the scheme localhost with an opaque path, and //host/ (#39000) as invalid. Both go to URL::parse as today.

Why https:// is not prepended for scheme-less strings, unlike parse_s3_endpoint: a registry string without a scheme has never been valid here (the env vars are ignored without http(s)://, and Scope::set_url falls back to the raw bytes), so there is nothing to keep working.

Open PRs in the same function: #39603 decodes the credentials with a decode_lenient_alloc of the same name and shape, so whichever lands second drops its copy. #39063 sends a username without a password as Basic; #38834 expands $ENV registry values; #40423 reworks .npmrc credential lookup.

Suites run with the debug build: config-precedence.test.ts, redacted-config-logs.test.ts, npmrc.test.ts, bun-publish.test.ts, bun-install-registry.test.ts, bun-install.test.ts, bun-add.test.ts.

The bun-install.test.ts run had 42 failures, all Git, GitHub or public-registry tests (DNSResolveFailed), identical in kind to the failures of the same run on main in this container.