fix(s3): version and lock objects written by other services like PutObject

Merged
VI
5d ago
+1763-757

Summary

Objects that other services write into S3 bypassed S3's write semantics: put_object_via_store and several direct writers built S3Objects by hand, ignored bucket versioning (no version id, overwrote the current object in place, no history), never applied the bucket's Object Lock default retention, and some were never persisted. On AWS an object delivered by another service is an ordinary new version.

One version commit for every writer. commit_object_version(_with) in objects/mod.rs is now the single path for every new object version: PutObject, CopyObject, CompleteMultipartUpload, replication and all cross-service deliveries.

  • Enabled bucket: new version id (or a caller-supplied one, e.g. a replica keeping its source's id; a re-written id keeps its history position and only becomes current if newest). A pre-versioning object is preserved as the null version.
  • Suspended bucket: null version semantics.
  • The bucket default retention is applied here, once, measured at commit (explicit lock headers from #2721 still win; a replica keeps its source's lock).
  • The body is written to the store before memory changes; on failure the error is returned and memory is left as it was (a preserved null relabel that already reached disk is mirrored in memory).

Cross-service writers (put_delivered_object): CloudWatch Logs exports/deliveries, ELB access logs, Athena query and notebook results, Firehose, DynamoDB export, ElastiCache snapshot export, ACM PCA audit reports, SES receipt-rule S3 action, the RDS aws_s3 export bridge, server access logs, inventory reports, CodeBuild artifacts. Side fixes:

  • Random or SHA-based quoted ETags replaced by the MD5 ETag (Firehose, Athena notebooks, ElastiCache, ACM PCA, SES, RDS bridge).
  • CodeBuild artifacts, inventory reports, Athena notebook results, ElastiCache exports, ACM PCA reports and RDS-bridge objects are now persisted.
  • Store failures reach the producer: Athena notebook calculations and ACM PCA audit reports now report FAILED, ElastiCache export returns an error, the RDS bridge returns 500, CodeBuild fails the build; DynamoDB export and Athena queries already failed.

Replication: replicas keep the source version id, get the replication time as LastModified, keep the source's lock or else take the destination default retention, and follow destination versioning (a pre-existing destination object is preserved as null; a suspended destination gets the null tag).

Follow-ups handled separately: replication of multipart-completed objects (not replicated on main either), replica handling of a source null version, and S3 event notifications / replication / default encryption for delivered objects.

Test plan

  • Unit (delivery.rs, service/tests.rs, inventory, replication): unversioned / enabled / suspended deliveries, default retention (Days, calendar Years, explicit lock wins), disk-store reload of every version and lock, store failure returns the error with memory unchanged (body write failing; sidecar rewrite failing), every version kind (replica, copy, MPU, delivery) gets the destination default, re-replicating an older version keeps the newer one current, MPU enabled then suspended, Athena notebook and ACM PCA failure reporting.
  • E2E: firehose_delivery_to_versioned_object_lock_bucket (two PutRecord calls -> two real versions, HeadObject version id, GetObjectRetention GOVERNANCE); s3, s3_persistence, firehose (incl. #2713), firehose_persistence, athena, dynamodb_persistence, elbv2_access_logs, logs_persistence, ses_receipt_actions pass locally. Container-backed suites (Lambda invoke via logs subscription, RDS aws_s3, ElastiCache) run in CI only: no Docker on the dev machine.
  • Docs: website/content/docs/services/s3.md Object Lock and "Other services -> S3" bullets.

Summary by cubic

Routes every new S3 object version — PutObject, CopyObject, CompleteMultipartUpload, replication, and cross-service deliveries — through a single commit path that applies bucket versioning and default Object Lock retention. Objects delivered by other services (CloudWatch Logs, ELB access logs, Athena, Firehose, DynamoDB export, ElastiCache, ACM PCA, SES, RDS bridge, server access logs, inventory, CodeBuild artifacts) now behave like ordinary PutObject writes on the destination bucket: real version ids on versioned buckets, the bucket default retention when no explicit lock is set, and persistence through the durable store. Replicas keep the source version id, take the replication time as LastModified, carry the source's lock when present, and follow the destination's versioning state.

Bug Fixes

  • Random or SHA-based quoted ETags replaced by the MD5 ETag for delivered objects.
  • CodeBuild artifacts, inventory reports, Athena notebook results, ElastiCache exports, ACM PCA reports, and RDS-bridge objects now persist across restarts.
  • Store failures reach the producer: Athena notebook calculations and ACM PCA audit reports now fail, ElastiCache export returns an error, and CodeBuild fails the build.