feat(firehose): deliver Xerial Snappy-compressed S3 objects

Merged
CA
4d ago
+804-14

Summary

Part 3 of the Firehose S3 delivery series tracked in #2708. Stacked on GZIP #2745 and its metadata-aware S3 writer: until #2745 merges, this diff also contains its commits, and the Snappy changes are the commits from feat(firehose): support Xerial Snappy delivery to S3 onward.

  • Implement the modeled Snappy compression format using pure-Rust raw Snappy blocks (snap 1.1.2, BSD-3-Clause, no dependencies, compression only) in the Xerial/snappy-java container.
  • Use .snappy, Content-Type: application/octet-stream, and Content-Encoding: snappy-java, preserving extension overrides, metadata, exact decoded bytes, and persistent restart behavior.
  • Fill 32 KiB input blocks across record boundaries, including delimiter bytes, as snappy-java's SnappyOutputStream does; each block is compressed independently. Test decoding is independent and bounded.
  • Include independently generated snappy-java fixtures, reproduction details, unit regressions, and official-SDK delivery/persistence/UpdateDestination coverage. Java is not a runtime or test dependency.
  • Keep immediate delivery unchanged; size/time buffering follows in its own increment.
  • Retain upstream #2730's default CRC64NVME full-object checksums, with assertions covering stored Snappy bytes (not decoded input), persisted checksum values, and SDK GET/HEAD checksum metadata across restart.

Why the Xerial container

Three incompatible Snappy containers exist. A published probe of real Amazon Data Firehose (floci-io/floci#2328, eu-west-1, 2026-08-15) found that CompressionFormat: Snappy writes .snappy objects with Content-Type: application/octet-stream, Content-Encoding: snappy-java, and the Xerial header 82 53 4e 41 50 50 59 00; snappy-java 1.1.10.8's SnappyOutputStream reproduced the AWS object byte for byte, while HADOOP_SNAPPY used Hadoop block framing with Content-Encoding: hadoop-snappy (still rejected at admission here). That probe delivered one 690-byte payload, so it confirms the header, metadata, and records sharing a block; the 32,768-byte block size for larger payloads follows snappy-java's default. The container bytes were also checked against snappy-java 1.1.10.8's SnappyCodec/SnappyOutputStream source. Compressed bytes may differ from AWS's (encoder heuristics), but decoded bytes and framing match.

Test plan

  • Verified on c5907ef3a (stacked on GZIP dfd71471c, upstream 774c772d6): 1,727 focused CloudFormation/Firehose/Logs/S3 unit tests, 86 rebuilt-server codec/admission/SES/checksum/persistence/encryption/write-side-effect SDK tests, one Athena regression, and 107 Firehose/SES/S3 conformance tests.
  • Workspace all-target Clippy, formatting, documentation counts, and diff checks passed on that revision.
  • A pre-publication code/design review checked the container bytes against snappy-java's source, re-verified the fixture and JAR SHA-256 values, and decoded the fixtures with a separate decoder; a STRIDE security review found no Snappy-specific issues (no untrusted Snappy is ever decoded; output is at most about 1.17× the input).
  • Unrestricted workspace testing of the GZIP predecessor stopped on the unchanged API Gateway fixed-port timeout (expected 502, actual 504 at closed 127.0.0.1:54321), independently reproduced on upstream. Snappy's final focused run does not claim that its workspace gate passes; no test exclusions were added. Revalidate after rebasing onto the merged predecessor before submission.

Series context

Refs #2708; does not close the tracking issue. Depends on #2745; branch feat/firehose-snappy is stacked on feat/firehose-gzip. Once #2745 merges, I will rebase onto main so only the Snappy commits remain and revalidate. Size/time buffering follows, stacked on this PR, in #2747.