fix(persistence): migrate pre-partition arn:aws: ARNs in persisted state

Merged
VI
22d ago
+2189-10

Summary

Closes #2610.

State persisted by a cn/gov/iso-region server before the ARN partition work (#2569..#2591) still held arn:aws:.... A one-time, load-time migration now rewrites it.

  • Mechanism: fakecloud_persistence::arn_partition::migrate_data_dir(data_path, region) runs in main.rs right after the version-file check, before any service loads, then sets arn_partitions_migrated = true in fakecloud.version.toml (new data dirs start with it set). A single marker instead of ~100 per-service schema bumps; old binaries ignore the field. Files are replaced atomically and the marker is written last, so an interrupted run re-runs idempotently.
  • Rules: regional ARNs (arn:aws:<svc>:<region>:...) take their region's partition whatever --region is (an arn:aws: ARN with a cn/gov/iso region can't be valid). Region-less ARNs (arn:aws:iam::..., arn:aws:s3:::...) take the server's partition, only when it isn't aws. AWS-managed policies (arn:aws:iam::aws:policy/...) stay as is, matching the managed-policy catalog lookup. Wildcards (*, ?) and ${...} variables move with the resources they match; region patterns follow the server's partition.
  • Coverage: values, map keys (ARN-keyed maps like SNS topics, tag maps) and ARNs inside stored documents (policies, templates, state-machine definitions; edited in place, formatting unchanged) in every <service>/*.json and logs/manifest.json, plus S3 bucket subresource files and object/multipart sidecars. A byte pre-scan skips files with nothing to fix, so commercial servers pay one scan.
  • Deliberately untouched: payloads with a stored hash/size (SQS bodies/attributes with MD5, ECR manifests, CloudWatch dashboards, EventBridge events); customer data (DynamoDB items/streams, SSM values, secrets, SES template content, SNS messages, S3 keys/user metadata/tags, log events); S3 bodies, log segments, container volumes. Unreadable paths (e.g. lost+found) are skipped individually; a failed write aborts startup so the marker is never set over stale state.
  • fakecloud-persistence now depends on fakecloud-aws (moved to layer 2 in scripts/publish-crates.sh; --check passes).

Edge cases (accepted): an aws-partition server still rewrites regional ARNs naming a cn/gov/iso region and sets the marker; no-marker dirs written by current main are treated as legacy (an IAM entity created on a cn server through a us-east-1-signed request would become aws-cn; indistinguishable); a migrated map key colliding with an existing new-partition key keeps the existing one and logs a warning.

Surfaces: new "ARN partition migration" section in docs/reference/persistence.md. No CLI/SDK change.

Test plan

  • Unit: 25 in arn_partition_tests.rs (rewriter rules, keys/values/documents, wildcards and variables, managed policies, idempotence, full data-dir runs, marker, regression test per review finding); persistence crate 97.
  • New e2e arn_partition_migration_persistence.rs: us-east-1 server creates KMS/SNS/IAM/SQS/S3/Lambda/DynamoDB state, snapshots rewritten to cn-north-1 with the marker stripped (realistic legacy dir), restart with --region cn-north-1 -> arn:aws-cn: from KMS DescribeKey/ListAliases (lookup by new ARN and alias), SNS ListTopics/GetTopicAttributes alongside a new topic, IAM GetRole + trust/inline docs + attached customer policies (managed policy still arn:aws:), SQS QueueArn + RedrivePolicy (body + MD5 unchanged), S3 bucket policy + object SSE-KMS key ARN (ARN-like key still reachable), DynamoDB DescribeTable, Lambda GetFunction, marker set. Fails with the migration disabled.
  • All 44 *_persistence e2e suites (139 tests).
  • fmt + workspace clippy clean; publish-crates.sh --check; local /code-review medium 4 rounds, last clean (fixed 11 findings incl. lost+found abort, stranded S3 objects, customer-data rewrites, wildcards/variables, escaped-ARN pre-scan miss, write errors marked migrated).

Summary by cubic

Migrates arn:aws: ARNs in persisted state to their region's partition, so a China, GovCloud, or ISO-region server restarted on older state stops mixing partitions and its ARN-keyed lookups (SNS topics, tag maps) work again.

  • Runs before any service loads, right after the version-file check, and records arn_partitions_migrated = true in fakecloud.version.toml; new directories start with the marker, and an interrupted run re-runs idempotently.
  • Regional ARNs take their region's partition, whatever --region is; ARNs without a region take the server's partition when it isn't aws, while AWS-managed policy ARNs keep the aws spelling to match the managed-policy catalog. Wildcards (*, ?) and ${...} variables move with the resources they match.
  • Rewrites values, ARN-keyed map keys, and ARNs inside stored documents in place, plus S3 bucket configuration and object sidecars; a byte pre-scan skips files with nothing to change, and ARNs behind a backslash escape (\narn:aws:...) in raw JSON/XML still count.
  • Unreadable paths are skipped per-file so the rest still migrates; a failed rewrite aborts startup so the marker is never set over stale state.
  • Leaves hash-verified payloads (SQS bodies/attributes with MD5, ECR manifests) and customer data (DynamoDB items, SSM values, secrets, SES template content, SNS messages, S3 keys, metadata and tags, multipart-upload tagging, client-supplied object headers, log events, container volumes) untouched.