perf(decofile): decompress .bin with node:zlib instead of npm:brotli

Merged
PE
15d ago
+42-5

Why

engine/decofile/fs.ts imports { decompress } from "npm:brotli@1.3.3" to read the production decofile (/app/decofile/decofile.bin). That package's index.js also does require('./compress'), which loads build/encode.js. That's an Emscripten build that allocates TOTAL_MEMORY = 318767104 bytes (304 MiB) when the module loads.

fs.ts is imported statically (engine/manifest/manifest.ts), so every pod of every site pays this, even though only decompress is ever called.

Measured on Linux (denoland/deno:2.5.6), in a script that only imports the package:

VmRSSDeno.memoryUsage().external
npm:brotli89 MiB312 MiB
node:zlib56 MiB3 MiB

So it costs ~33 MiB of real memory per pod, plus a 312 MiB "external" reading that makes memory dashboards and V8's external-memory accounting misleading. In a heap snapshot of a production-mode farmrio instance, that single Int16Array was 324 of 579 MiB.

What

Use brotliDecompressSync from node:zlib, Deno's native Rust brotli. No npm package, no WASM/asm.js heap.

Tests

  • New engine/decofile/fs.test.ts: writes a brotli + base64 decofile.bin (the production format) and reads it back through newFsProviderFromPath.
  • In a farmrio production-mode replay (800 real requests), external went from 309 MiB to 4 MiB with no functional change.

Summary by cubic

Swaps the decofile decompressor from npm:brotli to node:zlib, cutting per-pod memory usage. The old package loaded its Emscripten encoder at import time, reserving a 304 MiB heap in every pod even though only decompress was used; measured RSS dropped from 89 MiB to 56 MiB and external memory from 312 MiB to 3 MiB.

  • Replaces decompress from npm:brotli@1.3.3 with brotliDecompressSync from node:zlib in engine/decofile/fs.ts.
  • Adds a test in engine/decofile/fs.test.ts covering the production brotli + base64 .bin format.
  • Adds dispose on the fs provider to close its Deno.watchFs watcher and clear pending debounced reads, fixing a handle leak.
  • Starts the watcher in its own async task so a watchFs failure can't reject the decofile promise, and skips state updates after dispose.

Summary by CodeRabbit

  • Bug Fixes
    • Improved compatibility when reading Brotli-compressed decofiles, including Base64-encoded files, while preserving support for regular JSON files.
    • Improved cleanup when a filesystem provider is disposed, helping prevent lingering file updates after it is no longer in use.
  • Tests
    • Added coverage verifying that Brotli-compressed, Base64-encoded decofiles load and return the expected decoded content.