fix(logs): deliver raw gzip payload to Firehose subscription destinations

Merged
VI
14h ago
+61-4

CloudWatch Logs subscription filters with a Firehose destination passed encoded.as_bytes() (the base64 text of the gzip payload) as the Firehose record, so the delivered S3 object contained base64 text rather than gzip. In AWS the Firehose record is the gzip-compressed CloudWatch Logs payload itself, and the object is readable with zcat. Base64 is only the wire encoding for SQS/Lambda (and the Kinesis path already decodes it back to bytes).

Found while reviewing #2715. Adds a unit test asserting the record starts with the gzip magic and decodes to the DATA_MESSAGE payload.


Summary by cubic

Fixes CloudWatch Logs subscription filters delivering base64 text instead of the gzip payload to Firehose destinations, so delivered S3 objects now contain gzip data readable with zcat as in AWS.

  • Sends the raw compressed payload to Firehose; base64 remains only the SQS/Lambda/Kinesis wire encoding.
  • Adds a unit test asserting the record starts with the gzip magic and decodes to a DATA_MESSAGE payload.