perf(build): reuse kernel builds across worktrees; add a quick test-only kernel build

Merged
BO
2d ago
+308-17

A new worktree gets its native kernel in under a second instead of about two minutes when its kernel sources match a recent build, and a kernel edit can be tested after a 4 s rebuild instead of 90 s.

Step (16 vCPUs, other work running)BeforeAfterBenefit
npm run build:kernel in a new worktree, kernel sources unchanged~139 s, 595 MB target/0.8 s (restored from the cache)~170× faster, no target/
ensure-kernel before npm test, prebuild missing~139 s0.08 s~1,700× faster
Rebuild after a one-line resolver edit89 s (release)4.2 s (build:kernel:quick)21× faster
Cold build~139 s (release)83 s (quick)1.7× faster

One run each; these are single measurements, not a benchmark claim. The gaps are far outside run-to-run variation.

What changes

  • scripts/kernel-cache.mjs (new): a per-user cache of release kernel builds, keyed by the kernel source stamp that ensure-kernel already uses.
    • Only builds of committed sources are cached, and only plain release builds. A build with RUSTFLAGS or CARGO_PROFILE_* overrides is not cached.
    • It keeps the four most recently used entries per platform (about 82 MB each).
    • Location: CODEGRAPH_KERNEL_CACHE, else $XDG_CACHE_HOME/codegraph/kernel (%LOCALAPPDATA% on Windows); =off disables it.
  • scripts/build-kernel.mjs:
    • Host release builds restore from the cache before compiling and save after building. --no-cache compiles anyway.
    • --quick builds a new quick Cargo profile (no LTO, 16 codegen units, incremental). Its stamp is marked quick, and it is never cached.
    • CARGO_TARGET_DIR is honored. Before, the output path was hard-coded to target/.
    • The stamp is written only when the kernel sources did not change while cargo ran. Before, a file edited mid-build could leave a prebuild whose stamp did not describe its code.
  • scripts/ensure-kernel.mjs restores from the cache before compiling. It accepts a quick build of the current sources, with a reminder that a quick build is not for measuring.
  • npm run build:kernel:quick.
  • The build-validation skill gets refreshed timings and the --minWorkers=1 --maxWorkers=8 form; --maxWorkers alone errors in vitest 2.
  • Cross-target (--target) and release-workflow builds are unchanged: restore, save and --quick are host-only.

Verification (head 8320a704)

  • __tests__/ensure-kernel-stamp.test.ts, 23 tests. New ones use a fake cargo to cover build-kernel.mjs:

    • save then restore without compiling, and --no-cache;
    • quick builds never cached, with a marked stamp;
    • an edit during the build leaves the prebuild unstamped and uncached;
    • custom flags bypass the cache;
    • CARGO_TARGET_DIR;
    • restore from ensure-kernel without a toolchain;
    • no restore for uncommitted sources or with the cache off;
    • a quick stamp of other sources is rebuilt;
    • pruning keeps the four most recently used entries.

    The mid-build and custom-flag tests fail against the first version of this change and pass now.

  • Real builds: a release build saved to the cache; a second fresh worktree at the same commit restored it in 0.8 s, and the golden-dump tests passed on the restored kernel.

  • Full suite: 754 files passed, 40 tests skipped (vitest run --minWorkers=1 --maxWorkers=8). ensure-kernel restored a stale prebuild from the cache during one run and rebuilt-and-saved during the other.

  • Windows: the cache unit test is not gated by platform, but the fake-cargo tests are POSIX-only (shell-script cargo). Not run on Windows.

README rows checked: none apply (developer build tooling; no user-visible feature, command or quoted number).


Summary by cubic

Caches plain release kernel builds per user so a new worktree with unchanged kernel sources restores the prebuild in under a second instead of recompiling for ~two minutes, and adds an unoptimized quick build (npm run build:kernel:quick) that rebuilds after a one-line edit in ~4 s instead of ~90 s.

Details:

  • The cache is keyed by the kernel source stamp, keeps the four most recent entries per platform, and lives at CODEGRAPH_KERNEL_CACHE (or the platform cache dir; =off disables it).
  • Only plain release host builds of committed sources are cached; quick, cross-target, and RUSTFLAGS/CARGO_PROFILE_* builds bypass it, and build:kernel --no-cache compiles without it.
  • Both ensure-kernel.mjs and host release builds restore from the cache before compiling.
  • Builds with CARGO_TARGET_DIR now write there instead of hard-coded target/.
  • The prebuild stamp is written only when kernel sources did not change during the build, so a file edited mid-build leaves the prebuild unstamped and it gets rebuilt instead of carrying a wrong stamp.
  • ensure-kernel accepts a quick build of the current sources and warns not to measure performance with it.