fix(firehose): validate S3 compression formats before state changes

Merged
CA
2d ago
+1464-25

Summary

Closes #2711.

  • Validate the exact S3 compression enum (UNCOMPRESSED, GZIP, ZIP, Snappy, HADOOP_SNAPPY) at Firehose create/update admission. Unknown spellings and non-string values return HTTP 400 InvalidArgumentException; valid values and omitted-field semantics are preserved.
  • Cover classic/Extended S3 destinations and modeled nested S3/backup configuration and update members, using bounded explicit paths rather than a recursive scan of unrelated JSON.
  • Reuse the validator in the supported CloudFormation Firehose provisioning paths, rejecting invalid input before stream mutation.
  • Preserve the last successfully applied CloudFormation stack metadata when UpdateStack or UPDATE change-set execution fails. Review exposed that retaining the rejected template let an identical retry skip resource validation; regressions now cover both orchestration paths, retries, and subsequent successful updates.
  • Add unit, CloudFormation orchestration, and official-SDK coverage that compares complete stream state, versions, timestamps, stack templates, and parameters on rejected updates.

This standalone branch changes input validation, not codec delivery behavior. It does not implement compression codecs or expand CloudFormation's physical rollback engine.

Test plan

  • cargo test -p fakecloud-firehose — 34 unit tests passed.
  • cargo test -p fakecloud-cloudformation — 625 unit tests passed, including 7 Firehose/orchestration tests.
  • Server rebuilt from this branch, then Firehose API/persistence/compression-validation E2Es — 15 passed, including all four new SDK tests.
  • cargo test -p fakecloud-conformance --test firehose — 6 passed.
  • cargo test -p fakecloud-conformance --lib — 94 passed.
  • cargo clippy --workspace --all-targets -- -D warnings.
  • cargo fmt --all --check, documentation counts, and git diff --check.
  • Upstream CI-equivalent workspace test gate passed, with the single local WSL exclusion below.
  • Regression failures observed before fixes; independent review completed and its CFN retry finding resolved.

Local verification limitations

cargo test --workspace \
  --exclude fakecloud-e2e --exclude fakecloud-conformance \
  --exclude fakecloud-tfacc --exclude fakecloud-parity \
  -- --skip data_plane::tests::vpc_link_proxy_resolves_target_nlb_and_attempts_connection

The one excluded API Gateway test fails identically on untouched upstream 041ddc563 under this WSL network configuration (504 from a localhost timeout instead of its expected immediate-refusal 502). The exclusion is local verification only; no tests were disabled or conformance baselines changed.

An earlier unrestricted workspace attempt encountered Podman prerequisite failures, an unrelated RDS readiness timeout, and the two-hour tool timeout in Terraform acceptance tests. Those suites are separate from upstream's normal CI workspace test job. All affected Firehose/CFN E2Es and scoped conformance tests above passed.

Summary by cubic

Fixes Firehose accepting invalid S3 compression formats, and closes a CloudFormation retry hole that let identical rejected updates skip validation.

  • Validates CompressionFormat on Firehose create/update for classic and Extended S3 destinations, nested S3/backup blocks, and update members; invalid values return HTTP 400 InvalidArgumentException without mutating state. Redshift's intermediate S3 location additionally rejects Snappy and ZIP, which the Redshift COPY command cannot read.
  • Reuses the validator in CloudFormation provisioning, rejecting invalid input before stream mutation.
  • Failed CloudFormation updates retain the last successfully applied stack metadata and record resources already changed, so an identical retry still fails validation and a later valid update re-applies those resources instead of skipping them as unchanged.

Bug Fixes

  • Retained rejected templates previously let an identical retry skip resource validation, and partially applied changes were missed on the next update; regression tests cover both UpdateStack and change-set execution paths, retries, and subsequent successful updates.