test(install): serve GitHub tarball fixtures locally in the GHSA-pfwx-36v6-832x and lifecycle script tests

Closed
RO
2mo ago
+237-256

Problem

  • test/cli/install/GHSA-pfwx-36v6-832x.test.ts failed 5/5 on build 98240 with error: GET https://api.github.com/repos/jonschlinkert/is-number/tarball/98e8ff1 - 504; the same build failed both git dependencies also run preprepare, prepare, and postprepare scripts cases in bun-install-lifecycle-scripts.test.ts with the same 504 for dylan-conway/lifecycle-install-test.
  • Both tests install a real repository from api.github.com, so a GitHub outage fails them on every open PR. The GHSA file has done so since #27019 added it; the lifecycle cases predate it.
  • What the tests check (the integrity bookkeeping in src/install/extract_tarball.rs:47-79 and src/install/TarballStream.rs:1111-1182, and the prepare scripts of a GitHub dependency) needs a tarball with GitHub's layout, not GitHub itself.

Fix

  • test/harness.ts: githubTarball(rootDir, files) builds the tarball GitHub returns for /repos/<owner>/<repo>/tarball/<ref> with Bun.Archive: a <rootDir>/ entry first, then the files under it. bun takes the resolved commit from the archive's first entry (src/libarchive/lib.rs:1488-1508), which is why the root directory needs its own entry; the trailing-slash key is what produces it. isolated-install.test.ts carried an inline tar writer for exactly this layout and now uses the helper (net -29 lines there).
  • Both failing test files serve such a tarball from a local Bun.serve and pass its origin as GITHUB_API_URL, which alloc_github_url (src/install/PackageManager/runTasks.rs:1700) uses in place of https://api.github.com. This is the mechanism isolated-install.test.ts, bun-patch.test.ts and symlink-path-traversal.test.ts already use; the dependency specs are fake names, so a test that stopped going through the local server would fail instead of silently reaching GitHub.
  • The GHSA tests keep every previous assertion and add what a locally served tarball makes possible: the lockfile entry is compared as a whole against the sha512 of the bytes served, a rejected tarball must not end up in node_modules and must not replace the locked hash, and the tarball request count proves the cache-hit case downloads nothing.
  • Each GHSA case runs twice, against the buffered extractor and against the streaming one, because the two hash and verify independently and the old test (7.6 KB real tarball, below the 2 MiB streaming threshold) only ever exercised the buffered one. The fixture carries 600 KiB of incompressible filler so it exceeds the HTTP client's 512 KiB read buffer and always arrives in more than one chunk; with BUN_INSTALL_STREAMING_MIN_SIZE=1 that makes the streaming commit deterministic, and without it the default threshold keeps the buffered path. The --verbose lines [gh-dep] Extract / [gh-dep] Streamed assert which extractor ran.
  • The lifecycle fixture reproduces the real repository: a package.json whose six scripts run bun <script>.js, each writing <script>.txt, so the Blocked 6 postinstalls line, the #1234567 short-commit output and the six file checks are unchanged apart from the names.
  • Verified with bun bd test and with outbound network removed (HTTP_PROXY/HTTPS_PROXY unset in this container, which leaves only loopback): old GHSA file 0/5 pass, new file 10/10; old lifecycle cases 0/2, new cases 2/2; isolated-install.test.ts 66/66; the rest of the lifecycle file is unchanged (the three tests that fail here also fail on main in this container, they need node/bun on PATH).

Background

  • owner/repo#ref dependencies are downloaded as a tarball from the GitHub API rather than cloned. GitHub's tarballs unpack into a single <owner>-<repo>-<short sha> directory; bun records that directory name in the lockfile as the resolved commit (["dep@github:owner/repo#ref", {}, "owner-repo-sha", "sha512-..."]) and, since #27019, the sha512 of the tarball bytes after it, so a later install that has to refetch the tarball is checked against it.
  • bun install has two tarball extractors: bodies below BUN_INSTALL_STREAMING_MIN_SIZE are buffered and extracted in one go (extract_tarball.rs); larger bodies are fed into libarchive as chunks arrive (TarballStream.rs). The decision is made on the first body chunk (src/install/NetworkTask.rs:223-240) and needs a 2xx status, a body at least the threshold size, and more data still to come, which is why the fixture has to be larger than one socket read.

Still reaching GitHub after this PR

  • Through the same tarball endpoint, so convertible with this helper: the should handle GitHub URL ... cases in bun-install.test.ts, the git dependencies case in bun-install-registry.test.ts, bun-add.test.ts (#35149 is open for it with its own builder in dummy.registry.ts) and bunx.test.ts (#38467, likewise). Those PRs can import this export instead once it lands; the remaining files are left for follow-ups so this one stays a fix for the two files that went red.
  • Through git clone (git+https://github.com/..., SCP-style URLs, bun create), which needs a local repository rather than a tarball; #35035 covers the bun-add.test.ts instance.

no test proof · iteration 0 · Platform-specific test-only change; deferring to CI.