feat(firehose): deliver GZIP-compressed S3 objects

Merged
CA
4d ago
+2220-128

Summary

Part 2 of the Firehose S3 delivery series tracked in #2708, following merged #2715.

  • Encode each immediate-delivery batch as one real GZIP stream using the existing flate2 dependency. Preserve binary record bytes and apply AppendDelimiterToRecord before compression.
  • Use .gz by default, honor FileExtension overrides, and persist Content-Encoding: gzip together with the encoded body and its MD5 ETag.
  • Add a metadata-aware shared S3 writer without changing existing callers. Delegate to upstream's versioned delivery commit so bucket versioning, default Object Lock retention, ownership, and durable failure handling remain intact.
  • Cover official Firehose/S3 SDK calls, raw CloudWatch Logs payloads through the internal adapter, and exact compressed bytes and metadata surviving restart, including bucket-default KMS, replication, and S3 notifications.
  • Update service documentation for this increment only. Delivery remains immediate; Snappy and size/time buffering follow separately.
  • Follow the owner's #2708 admission convention: modeled configurations still round-trip, but requesting an unimplemented delivery capability fails public record admission with InvalidArgumentException naming the setting. Preserve CFN settings so those streams get the same checks rather than silently writing substitute bytes.
  • Derive the output codec from the same parser admission uses (Compression::from_format), so admission and encoding cannot disagree and no request path relies on an unreachable!.

Rebase compatibility

Rebased onto upstream main at 774c772d6. Retains #2712's SES publishing, #2713's source admission, #2714's compression validation, #2720's raw GZIP Logs records, #2725's ordinary S3 versioning/Object Lock semantics, #2730's default CRC64NVME full-object checksums, and #2737's shared write-side-effect path. Checksums and ETags describe the compressed S3 payload, not decoded records or at-rest KMS ciphertext. Bucket-default encryption, replication, and notifications remain on the shared commit path.

Failed persistent writes do not publish memory-only objects, matching current upstream. Records are still acknowledged without retries in this immediate-delivery increment; the buffering increment adds retention/retry of failed batches.

Test plan

  • Verified the final tree (head dfd71471c): 1,721 focused unit tests across CloudFormation (635), Firehose (70), Logs (361), and S3 (655); 82 rebuilt-server SDK tests covering Firehose/GZIP, compression/admission validation, SES events, checksums, persistence, SSE-KMS, and write side effects; one Athena cross-service regression; 107 Firehose/SES/S3 conformance tests.
  • Workspace all-target Clippy with warnings denied, formatting, documentation counts, and diff checks passed on that tree.
  • cargo test --workspace was run without exclusions on 055708266 (before the final single-parser refactor and docs wording) and stopped on fakecloud-apigateway::data_plane::tests::vpc_link_proxy_resolves_target_nlb_and_attempts_connection: expected 502, got 504 (128/129 API Gateway tests passed). The exact failure was independently reproduced on untouched upstream 153774026; API Gateway's test and implementation are unchanged by this PR or the later upstream commits. Closed port 127.0.0.1:54321 times out in this WSL environment instead of refusing connections. This is a failing local workspace result, not a passing gate; no test skip/exclusion was added. The focused gates above passed on the final revision.
  • A pre-publication code/design and STRIDE security review of the whole stack found no GZIP-specific vulnerabilities; its codec-parsing and documentation findings are addressed here.

Maintainer alignment

Follows the owner's #2708 guidance: exact byte concatenation, modeled-enum acceptance on Create/Update/CFN, and named InvalidArgumentException on unsupported record admission. The interval-range correction and Firehose-owned flusher/reset/shutdown lifecycle remain in the later buffering increment, not this codec PR.

Series context

Refs #2708; does not close the tracking issue. The next increments are stacked on this PR: Xerial Snappy in #2746, then the complete buffered-delivery lifecycle in #2747.