#4101 fix(plugin-state): transact writer mutations before rollout

Summary (AI generated)

The first writer-only production trial of #4038 exposed replayed queue-mutation results. Fifteen distinct Worker invocations reported 112 claims within about 77 seconds, while the confirmed origin had allocated only eight claim generations. A later pg_stat_statements inspection showed eleven actual claim executions. Each successful claim must allocate one generation.

Run sync claim/ACK, garbage-collection claim/ACK, and explicit repair enqueue in separate short read-write transactions. Commit each before KV/R2 I/O. Keep shared Hyperdrive caching, public plugin reads, generation fencing, retry ceilings and provider budgets unchanged.

Select a newly provisioned KV namespace and R2 pointer bucket for the API and all nine production plugin Workers. Nonproduction bindings are unchanged. These fresh resources are required because the writer was paused and source changes during that gap were not queued.

Motivation (AI generated)

These custom PostgreSQL functions mutate leases and progress even though their callers issue SELECT statements. Replaying a cached claim makes the drain process already-settled leases, inflates its progress counters, and slows backfill. The regression tests model an outside-transaction cache and verify that successive leases and GC pages actually advance.

Business Impact (AI generated)

This fixes a blocker found before enabling any KV device reads. All production read flags remain off, the writer is paused, and replicas remain available. No response-time improvement or cost saving is claimed from this patch; those remain production rollout gates.

Test Plan (AI generated)

• Backend lint and backend TypeScript checks.
• 113 unit tests across 12 files, including plugin-state and resource-isolation coverage.
• Nine focused source tests cover successive mutation leases, GC cursor progress, explicit repair enqueue, and committing before provider I/O.
• Independent review found no blocking defect and independently passed 13 source/isolation tests.
• Production API Wrangler dry-run build.
• Parsed configuration verifies one matching fresh KV/R2 pair across the API and all nine production plugin environments, with nonproduction unchanged.
• CI on 0dc8fb21d6: all 46 latest checks are complete, with 35 successful and 11 skipped. The Cloudflare Workers shard 1/2 retry passed on unchanged code after the original Docker port-bind failure before tests.
• After deployment, compare real origin claim executions/generations with actual queue progress.
• With old drains stopped, reset namespace-specific GC cursor/lease and backfill bookkeeping, explicitly enqueue full repair, and verify coverage, archives, leases and purges before restarting reads.
• Verify the deployed snippet KV-learning guard, usable regional cohort telemetry and real source-change propagation before the 0.1% read canary.

No migration, permissions, primary request-path reads, global query-cache settings or queue retry budgets are changed.

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by CodeRabbit

• Bug Fixes
  • Improved the reliability of plugin state synchronization, cleanup progress, and repair requests when operations overlap or cached results are involved.
• Documentation
  • Clarified how state-changing operations are committed and how they relate to storage work.