[Superseded] Combined Kafka and account-erasure release

Closed
AS
5h ago
+32404-6114

Superseded combined effort

The user requested two independent PRs and Paseo workspaces on November 8:

  • #3233: Kafka broker migration, branch/workspace kafka-broker-migration.
  • #3234: Account-erasure hardening on Redpanda, branch/workspace account-erasure-hardening.

Both start directly from main. They currently contain scoped extraction plans; runtime extraction is still pending. Preserve this draft at ed26f38b8 as source/evidence reference. Do not merge or deploy this combined branch or execute its old cutover packet. Production remains on Redpanda. Prior cleanup and release receipts remain historical records; no new production action is authorized by this split.

Summary

Move metric-stream sensor transport to KafkaJS on Kafka protocol (v2 topics, KAFKA_BROKERS), retire Pulsar/Bookie/Node r2-archive dual paths, and keep erasure fail-closed behind the existing stock_physical_maintenance_ack until Kafka tiered-storage (R2) physical absence is operator-qualified.

Draft: do not merge or deploy until cutover gates below are green. Automatic Deploy Web remains disabled.

Intended release scope

  • KafkaJS producer/consumer/quarantine + drain; domain jobs use the Kafka publisher factory.
  • Exact v1 topic produce refuse; legacy consume only where explicitly retained.
  • Connect/R2 archive CAS retained as the JSONL archive path; dedicated R2 Kafka-TS bucket for tiered segments (stock Aiven RSM + Cloudflare R2 IaC).
  • Account erasure: request-owned identifiers, consumer drain quarantine HWM (topic/partition/low/offset retained through PII scrub), PeerDB cutoff, ClickHouse/Postgres erasure, and day-29 retention verification.
  • Physical gate stays operator stock_physical_maintenance_ack (ingest_fence.completed_at ≤ verifiedCutoffAt ≤ verifiedAt ≤ now); TTL/HWM alone never satisfy TS cold-object absence.

Explicitly out of this merge

  • Claiming Kafka-TS physical absence from log-start / remote metadata alone.
  • Custom segment-body parsers or a new certificate framework.
  • Cloud/native TS qualification labs, mutation forks, or broader runtime instrumentation until Root approves the stock physical-absence plan.
  • Analytics journals/placements beyond existing PeerDB CDC.

Local evidence (Foundation)

  • Unit slice (SRC): green prior KafkaJS cutover.
  • Focused real Kafka/PG/CH erasure integ (2026-11-06, --maxWorkers 1): 78/78 across kafka-consumer, clickhouse-sink, kafka-drain, clickhouse-erasure, postgres-erasure, process-account-erasure-request, account-erasure scrub (a947534ca keeps quarantine HWM after PII scrub).
  • Local TS disabled in that slot — no cloud/native TS qualification claim.

Merge and activation gates

  • Root-reviewed Kafka-TS physical-absence/ack plan (stock RemoteLogManager retention/delete + R2 inventory/all-version cleanup under existing ack).
  • Required CI/review gates including mutation; no exclusions or threshold reductions.
  • Production Infisical/KAFKA_BROKERS, TS R2 IaC, host budget, maintenance approval, whole-host soak.
  • Keep unmerged until cutover is ready.