SOW-0160: Go writer declares a 256 KiB XZ dictionary

KT
1d ago
+668-311

Stacked on #19. Completes SOW-0160 (.agents/sow/done/SOW-0160-20261015-go-xz-reader-dictionary-allocation.md).

Decision

  • User decision D1 = A (2026-11-03): the Go writer declares a 256 KiB XZ dictionary, the same value the Rust writer uses.

Why

  • Every XZ reader allocates the dictionary size the writer declares, once per DATA decode. The Go writer used the library default of 8 MiB, so a 600-byte payload cost an 8.4 MB allocation.
  • The reader can't lower this: ulikunitz/xz v0.5.16 uses the larger of the reader's cap and the stream's declared size, and it has no reset API for reusing a decoder.

Changes

  • go/journal/writer_compression.go: xzDictCap = 256 << 10 is now set in xzCompress.
  • New go/journal/writer_compression_xz_test.go:
    • A byte-level check that the declared dictionary is 256 KiB, with round trips of payloads up to 4 MiB (two of them larger than the dictionary).
    • A decode benchmark comparing 256 KiB, 1 MiB and 8 MiB declared dictionaries.
  • Spec (product-scope.md) and docs/Options-Reference.md record the declared dictionary.

Results

Go decode, 600 B payload256 KiB8 MiB
time0.24-0.28 ms1.6-2.0 ms
allocated0.30 MB8.42 MB

Test file: 5,000 Go-written XZ rows, exported with journalctl --file F -o export:

Reader8 MiB file256 KiB file
Rust journalctl0.64-0.77 s user, 13 MB RSS0.06 s user, 5 MB RSS
Go journalctl22.5-27.2 s user4.6-4.8 s user
stock journalctl v2620.11-0.12 s0.12 s (unaffected)
  • Exported output is byte-identical between the two files for every reader.
  • Files written before this change still declare 8 MiB and still read.

Validation

  • go test ./... and cargo test --workspace pass.
  • Compression matrix for xz, zstd and lz4: 72/72 pass.
  • Byte identity passes.
  • Go and Rust writers both declare 256 KiB (parsed from the matrix files).
  • SOW audit is clean.

Summary by cubic

The Go XZ writer now declares a 256 KiB LZMA2 dictionary instead of the library's 8 MiB default, matching the Rust writer. Reads of new Go-written XZ files cost the Go reader ~6x less time and ~28x less allocation per small decode, and the Rust journalctl runs ~10x faster; decoded content is unchanged and older 8 MiB files still read.

  • One constant in xzCompress sets the dictionary; no other writer path or dependency changes.
  • New tests parse the declared dictionary from the stream header, round-trip legacy 1 MiB and 8 MiB streams, and add a decode benchmark.
  • Seal tests in both SDKs skip when journalctl was built without sealing support, so negative seal tests cannot pass for the wrong reason.
  • Measured results refute the earlier inference that stock journalctl would benefit: liblzma decodes 8 MiB and 256 KiB files at the same cost.