feat(plugin): serve plugin state from KV with progressive rollout
Summary (AI generated)
Cost objective: serve plugin reads from the edge, retire the regional read-replica fleet after full rollout, and retain at most one fallback replica if needed. The primary database remains the source of truth. Zero read replicas is the eventual option once every remaining SQL dependency has a safe replacement.
[!NOTE]
Production rollout remains disabled. Current head e58e04aaec is ready for human review: CI has no failed or pending checks, CodeRabbit approved this exact head, and all 28 threads are resolved. All 30 PostgreSQL contracts and Cloudflare db/kv-1/kv-2 tests passed. The Node 24 SDK upgrade remains separate in #4089. Worldwide latency, freshness and net savings remain production rollout acceptance gates; merging this PR does not enable its flags or retire replicas.
Add an opt-in KV + R2 + Cache API path for updates, manifest sizes, stats and channel self-service, preserving response logic and regional Worker placement. Keep the existing replicas during validation, then reduce the fleet after the retirement gates below pass. KV answers cannot teach shared DB/snippet caches; the original DB manifest-size cache remains untagged on v1. KV-only entries use separate keys/tags.
R2 and KV reads have separate 50 ms deadlines; requests are capped at 16 distinct state-document keys with four concurrent document loads. Oversized key fanout uses the existing batched replica path before sink writes. Percentage cohorts exclude requests without device IDs.
Writer ACKs purge only the isolated plugin-state tag, without the replica-lag repeat ladder. Content digests skip redundant purges; retries and explicit full resets purge conservatively if an earlier publication may have missed its ACK. Every publication still advances its generation fence. Full jobs take priority over backlog. Unrelated ACKs preserve repair cursors in backoff, explicit repair clears prior completion markers, and older ACKs cannot falsely complete a newer repair. Continuations persist atomically under live leases, cooldown claims report bounded retry delays, and exhausted full scans cannot restart through ordinary source changes. Ordinary claims during full-repair backoff publish version/device keys and the current app snapshot without changing or restarting the saved cursor or claiming full completion. Archived scans still allow current app snapshots, channel/plan changes and app-deletion tombstones to publish without a history scan; missing independent dependencies fall back to replicas.
Measured: most local warm request types are faster; warm manifest maps are essentially unchanged; new-bundle publication was slower in one observation. Cost target: eliminate the retired replicas' compute, disk and replication-transfer charges. Keeping the fleet during the canary is a temporary migration cost, not the intended end state. The final net saving depends on the optional retained replica and the measured replacement workload.
Performance comparison (AI generated)
Measured November 10, 2026 at 04:48 UTC on the canary-isolation and repair-recovery implementation committed as dcd6f5e404, using local workerd and PostgreSQL on one machine, timing-log sampling off. Paired requests check status/JSON and DB/KV provenance. These are local wall-clock timings, not worldwide network latency or billed CPU. They predate the subsequent archived-app, repair-backoff, explicit-enqueue and unpublished-app fallback fixes; those paths are covered by regression tests rather than timed by this run.
The separate CodSpeed CPU suite reported CRC32 and semver regressions on c453b15ae4, while warning that base/head hardware, software and SDK major versions differ. An independent source audit confirmed both benchmark bodies and all CLI implementations are byte-identical to main; neither exercises KV/R2. The report is therefore inconclusive. Head eeacef7430 restores SDK 5.7.1 to match main, and moves the supported Node 24 SDK 6.1.1 upgrade to #4089. SDK 5.7.1's untested-Node-24 warning remains a known limitation until that upgrade and a new baseline land. The CodSpeed 6.0 release documents a one-time simulation calibration change. Once main has its SDK 6 baseline, refresh this PR with matching settings. No thresholds, benchmark cases or regression acknowledgements were changed. The HTTP timing tables below remain independent historical workerd measurements; the CPU check does not measure the new KV request path.
Warm requests — 100 samples per path (AI generated)
| Scenario | DB p50 | KV/R2 p50 | DB p95 | KV/R2 p95 |
|---|---|---|---|---|
| Existing bundle, up to date | 13.22 ms | 3.53 ms | 16.71 ms | 4.63 ms |
| New bundle available | 13.26 ms | 3.59 ms | 15.66 ms | 5.18 ms |
| Channel list | 16.24 ms | 3.87 ms | 22.84 ms | 5.24 ms |
| Current channel | 15.46 ms | 4.37 ms | 19.65 ms | 5.79 ms |
| Stats | 10.31 ms | 2.65 ms | 14.79 ms | 4.98 ms |
| Stats batch | 16.06 ms | 3.23 ms | 19.39 ms | 4.56 ms |
| Manifest sizes | 1.29 ms | 1.30 ms | 2.97 ms | 1.54 ms |
After cache-tag purge — 10 samples per path (AI generated)
Each cohort starts after a tag purge, including the new isolated KV tag. Untagged DB caches, including day-long v1 manifest-size maps, remain warm. Provider caches and database buffers also remain warm. This is not a complete Cache API cold test. With 10 samples the raw nearest-rank p95 equals the maximum, labeled max below.
| Scenario | DB p50 | KV/R2 p50 | DB max | KV/R2 max |
|---|---|---|---|---|
| Existing bundle, up to date | 12.75 ms | 3.81 ms | 16.14 ms | 5.17 ms |
| New bundle available | 14.88 ms | 3.87 ms | 55.21 ms | 5.63 ms |
| Channel list | 15.35 ms | 4.66 ms | 18.57 ms | 7.16 ms |
| Current channel | 10.48 ms | 4.96 ms | 12.37 ms | 6.60 ms |
| Stats | 14.97 ms | 4.26 ms | 16.43 ms | 5.73 ms |
| Stats batch | 20.11 ms | 5.01 ms | 45.23 ms | 7.75 ms |
| Manifest sizes | 1.28 ms | 3.83 ms | 1.45 ms | 5.60 ms |
The tag-purged KV manifest map is slower than the DB map that stays warm. Warm manifest-map latency is essentially unchanged.
New bundle publication to first matching response (AI generated)
| Path | Channel-promotion COMMIT → matching response | Polls |
|---|---|---|
| DB | 19.01 ms | 1 |
| KV/R2 | 72.09 ms | 3 |
KV/R2 publication was slower in this observation. One observation per path, 25 ms polling plus request time, after upload preparation. This invokes the writer drain and local purge manually while polling; it does not benchmark production trigger/queue wakeup, replica lag or Hyperdrive caching. Faster warm reads do not prove release-freshness parity.
Channel surfing — single observations (AI generated)
Independent fresh devices per mode follow the same SET/read/DELETE/read lifecycle. These are individual operations, not latency distributions.
| Operation | DB | KV/R2 |
|---|---|---|
| 7.0.0 SET beta | 22.83 ms | 5.68 ms |
| 7.0.0 GET current after SET | 16.24 ms | 3.76 ms |
| 7.0.0 DELETE override | 23.72 ms | 4.99 ms |
| 7.0.0 GET current after DELETE | 18.72 ms | 8.35 ms |
| 7.34.0 SET beta | 44.94 ms | 4.54 ms |
| 7.34.0 GET current after SET | 19.69 ms | 4.15 ms |
| 7.34.0 DELETE override | 15.95 ms | 3.90 ms |
| 7.34.0 GET current after DELETE | 23.29 ms | 4.33 ms |
The fixture has fewer than 500 forced overrides and fitting manifests. Large override sets, unknown version-name fallbacks, oversized manifests, pointer/KV timeouts, production wakeups and cross-region visibility require separate measurements. Keep replicas and placement until regional latency and freshness pass.
Full method, validation and raw measurements
Price comparison (AI generated)
Historical Google baseline — actual billing-export data (AI generated)
The original investigation captured daily billing rows on November 6. The November 1–5, 2026 window averages approximately €22.62/day, or €679 for a 30-day run rate. This is a historical gross-cost baseline before credits/tax, reconstructed from rounded daily export rows; it is not a fresh invoice or a measured post-rollout bill. It includes the November 3 replication-transfer spike.
| Google cost component | Historical average/day | 30-day run rate | Target after replica retirement |
|---|---|---|---|
| Replica/subscriber compute | ~€13.62 | ~€409 | Eliminate compute for retired instances; retain only the optional fallback instance's cost |
| Disk | ~€3.57 | ~€107 | Eliminate disks attached to retired instances, subject to deliberate snapshot/retention choices |
| Inter-region replication | ~€3.77 | ~€113 | Eliminate replication streams to retired instances; any retained replica still has its own stream |
| Query-result transfer to Cloudflare | ~€1.50 | ~€45 | Reduce with actual KV serving coverage; remaining SQL fallbacks and source publication traffic must be included |
| Backups / other exported costs | ~€0.15 | ~€5 | Audit attribution and retention; do not assume every shared/exported charge disappears |
| Total | ~€22.62 | ~€679 | Target: remove most of the regional fleet bill, retaining only the optional fallback and remaining attributable costs |
The same export recorded an October 28 replication spike of €47.37 in one day. Keeping every replica during validation temporarily retains this exposure. Retiring replica instances and their replication streams is the step intended to remove the corresponding recurring and resync costs. Google documents separate instance charges and cross-region replication-log transfer charges. Cloud SQL replication billing.
| Deployment stage | Google component, using the historical workload as a reference | Cloudflare component |
|---|---|---|
| Before migration | ~€679/month regional-fleet run rate | Existing Worker costs; no enabled plugin-state read/sync workload |
| Temporary validation: KV plus all existing replicas | ~€634–679/month only if result egress falls by 0–100% and other Google costs stay fixed | Added KV/R2 reads, publication, GC, storage and incremental Worker work |
| Intended production target: 100% KV eligibility, regional replicas retired, one fallback retained | K + G, rather than the full regional-fleet bill | C: measured incremental KV/R2/Worker cost below |
| Eventual zero-read-replica target, after replacing all SQL fallback dependencies | G; recurring instance/disk/replication charges attributable to retired replicas are removed | C, including the cost of any additional SQL-free fallback/availability design |
The savings objective is replica-fleet retirement. The approximately €45/month result-egress opportunity applies only while the full fleet is kept. It is not the ceiling on the final architecture's savings. Retiring the regional instances also targets their compute, disk and replication charges.
Here K is the fully loaded cost of the one retained replica, including its compute, storage, replication and fallback traffic; G is any remaining attributable Google/source/retention cost; C is the incremental Cloudflare serving/publication/cleanup/storage/Worker cost. The primary database's normal bill remains separate and is not being eliminated. Compare the historical fleet baseline with K + G + C (or G + C with zero read replicas) after using one documented billing currency. An exact one-replica forecast needs the chosen instance's size, region, replication volume and measured fallback traffic; dividing the aggregate bill by the number of regions would be misleading.
The implementation in this PR is the migration foundation. Setting PLUGIN_STATE_KV=on selects all devices to try KV; it does not mean every response is served without SQL. Replica retirement follows the measured routing/coverage stages below. Actual net savings are not yet measured, so the target is not presented as an achieved bill or a fixed $10–40 replacement estimate. EUR billing and USD list prices remain separate until an exchange rate is explicitly applied.
Cloudflare public unit prices — verified November 10, 2026 (AI generated)
| Resource | Included monthly allowance | Additional usage, USD |
|---|---|---|
| KV keys read | 10 million | $0.50/million |
| KV keys written / deleted / list requests | 1 million of each | $5.00/million of each |
| KV stored data | 1 GB | $0.50/GB-month |
| R2 Standard pointer GETs (Class B) | 10 million | $0.36/million |
| R2 Standard pointer PUTs / lists (Class A) | 1 million | $4.50/million |
| R2 Standard storage | 10 GB-month | $0.015/GB-month |
Allowances are shared with existing account usage. R2 rounds aggregate usage upward to billing units; the following linear estimates are unit economics, not exact invoices. Contract rates, taxes, unused allowances and existing consumption can change the bill. Pricing GB is 10^9 bytes, not GiB.
Sources: Workers KV pricing, R2 pricing and rounding.
Added read-cost scenarios (AI generated)
A live document missing from Cache API requires 1 R2 GET + 1 KV GET, or a linear $0.86 per million missed documents. A Cache API document hit avoids those storage operations. Let R be eligible requests/month, d the average documents requested per request and h the document Cache API hit fraction:
Added read-operation cost ≈ R × d × (1 − h) / 1,000,000 × $0.86
| Assumed document hit rate | Documents/request | Linear USD per 1M eligible requests | Linear USD at 1.32B eligible requests/month |
|---|---|---|---|
| 99% | 1 | $0.0086 | $11.35 |
| 99% | 2 | $0.0172 | $22.70 |
| 90% | 1 | $0.0860 | $113.52 |
| 90% | 2 | $0.1720 | $227.04 |
| 0% (stress reference) | 1 | $0.8600 | $1,135.20 |
The 1.32B scale is an illustrative 44M Worker requests/day × 30, close to the original November 6 traffic observation. It assumes all modeled requests are KV-eligible with no fallback/retries; these optimistic hit rates and document counts are not measured production forecasts. The existing snippet reduces how many requests reach Workers. The row costs exclude writer/GC/storage and existing Worker costs. Missing pointers and timeouts have different operation counts and can still add DB fallback traffic.
Writer, cleanup and shared costs (AI generated)
- An uncontended live document publication does KV PUT + R2 GET + conditional R2 PUT: $9.86 per million document publications at linear rates. An existing app-document refresh also reads its previous R2/KV document: $10.72 per million app refreshes, before extra manifests/documents. One release can publish many documents.
- GC adds an R2 GET for each eligible candidate examined, including retained current values: $0.36/million candidates. Obsolete values add $5/million KV deletions; KV listing adds $5/million list calls. Repeated scans, retries and initial backfill must be counted.
- Historical tombstone fences remain in R2; full repairs still refresh them. Full-repair operations and pointer storage grow with historical key count. The operation budget bounds each invocation, not total lifetime cost.
- Add KV/R2 stored bytes, source DB traffic, purge work, observability and changed Worker CPU/writer invocations. Cache isolation can reduce shared-cache refills as the KV share rises, increasing Worker invocations after cache expiry; measure that incremental traffic at every regional share increase. The document-read scenarios above do not model it. Many Worker requests already exist in both architectures: do not count the entire shared Worker bill as a new KV cost or assume it disappears. Standard list pricing is $5/month with 10M requests and 30M CPU-ms included, then $0.30/M requests and $0.02/M CPU-ms; actual account/contract billing must be checked. Wall-clock benchmarks cannot infer billed CPU. Workers pricing.
- Production measurements must record document hit rate, documents loaded/request, R2/KV operation counts, published/collected documents, fallback bytes, stored bytes, source traffic and CPU. A fixed 30-second document TTL makes 90%/99% hit scenarios optimistic for cold app/location pairs. Replica fallbacks for unpublished app IDs and unknown version names, plus repeated GC inspections, add costs: 400 eligible pointer GETs every 10 seconds would be 103.68M/month or $37.32 at linear list rates before allowances/rounding; this is an illustration, not measured throughput. Savings are the change in the full Google + Cloudflare bill, not just the KV read unit price.
Partial rollout and recovery (AI generated)
The rollout ends with retirement of the regional replica fleet, with an optional single fallback replica. Keeping every replica is the validation stage. The read share controls stable app/device eligibility among requests reaching one regional Worker; it does not describe the percentage of all edge traffic actually answered by KV. Existing upstream DB/snippet answers can short-circuit eligible devices. Measure actual X-Plugin-State provenance.
-
Deploy the snippet's KV-learning exclusion first. Apply the migration and deploy API/plugin workers with read and sync switches off; verify the API handles the new isolated purge scope.
-
Establish primary load, replica QPS/result bytes, purge queue/API rate and existing edge/snippet-hit baselines before enabling writer/backfill. The read percentage does not bound those background workloads. Statement triggers/configuration checks still have deployment overhead with sync off.
-
Finish/verify backfill and explicit failed-key repairs. Start at
PLUGIN_STATE_KV=0.1%in one regional Worker, authoritative and strict both false, replicas intact. Compare errors, p95/p99, cold reads, fallbacks, and real publish/channel/override/delete/plan visibility with the DB control cohort. -
At each regional share increase, including
on, measure snippet on-prem/plan-block/edge-answer hit rates, app-status/read-cache hit rates, Worker invocations per incoming device request, latency, provider operations and CPU over a window that includes cache expiry. KV answers cannot refill those shared caches; DB-cohort and fallback answers can. Shrinking DB traffic may therefore increase Worker traffic and cost. Complete coverage alone is insufficient: stop or roll back if latency or cost gates fail. Read rollback isPLUGIN_STATE_KV=offin every enabled region; it stops new KV sessions after configuration applies, not in-flight requests. -
At
PLUGIN_STATE_KV=on, complete a production soak with actual KV/replica provenance and per-region latency/freshness measurements. Classify and reduce the remaining SQL reads: unpublished/unknown app or version IDs, missing historical manifests, devices without published override keys, oversized manifests/batches, provider timeouts and query escapes. Keep the primary out of plugin request-path reads. -
For the one-fallback-replica target, explicitly remap every regional read route/binding to the retained replica before retiring an instance. Current routing only switches an existing lagging regional route to EU; simply deleting a regional binding can leave no permitted read route and fail requests. Include plugin
/latency, replica monitoring, schema/deploy tooling and other real consumers in the audit. Measure retained-replica capacity and remote-fallback p95/p99 in every region; satisfy the latency target by improving edge coverage/availability where needed. Retire regional instances in stages after each gate passes. Establish a tested rollback using the retained capacity or an explicit replica-restoration plan; the old regional DB rollback is no longer instantly available after deletion. -
For zero read replicas, first replace every remaining SQL-dependent fallback and the plugin SQL
/latencyprobe with a safe non-primary serving/health design, including publication gaps, large/historical manifests and provider-failure behavior. Authoritative mode and strict mode do not implement that replacement. Remove the final read replica only after these dependencies, correctness, availability, freshness and per-region latency gates are met.
Before enabling authoritative mode, measure requests for app IDs without a published pointer, including IDs that never existed. They always use replicas even when the missing pointer is cached; include their QPS, egress and fallback latency in the cost and load gates.
Regional tooling requires explicit environment names and blocks regional plugin-state settings in shared production bulk env files. These examples are dry runs:
bun run plugin-state:set 0.1% prod_eu --dry-run
bun run plugin-state:timing 1% prod_eu --dry-run
bun run plugin-state:set off prod_eu --dry-run
Writer pause is a separate recovery operation. Turn reads off everywhere before disabling sync. Source changes while disabled are not queued; daily app refresh cannot repair every device/version/manifest key. After an untracked source gap, use a fresh KV namespace and fresh pointer bucket, consistently bound to API/plugin workers, drain old invocations and purge/expire old KV Cache API entries, then explicitly enqueue full repair and verify newly populated full_synced_at markers, drained full cursors and an empty failed-work archive before restarting reads. An empty queue or an old completion marker alone is insufficient. A full sync in the old namespace alone cannot discover every hard-deleted historical manifest. Explicit repair atomically enqueues resets and clears prior completion markers, refreshes exhausted pending budgets without releasing live leases, and includes deleted apps known only by pending work. Known archived keys can be repaired explicitly; automatic daily poison retry resets are removed. Archived apps are excluded from daily refresh, so a lost notification can leave valid stale state until KV reads are disabled and repair succeeds. A deleted app's tombstone prevents serving it, while its other keys remain until explicit cleanup. A manual reset overlapping a live empty-ref full job can perform one redundant scan; budget that maintenance cost. Valid-but-stale documents are not universally detected by the reader.
Detailed rollout, pause and restore procedure
Motivation (AI generated)
Replace regional database reads with precomputed edge state so the regional read-replica fleet can be retired after a measured rollout, optionally retaining one fallback replica. Preserve endpoint contracts and worldwide latency throughout that migration. The current implementation provides the staged read path; final cost and global publication freshness are production acceptance gates.
Business Impact (AI generated)
The intended business result is a substantially smaller database-serving footprint: retire the regional read replicas after KV adoption, keeping at most one fallback replica if necessary. This removes the corresponding compute, disk and replication costs, rather than limiting the benefit to result-egress reduction. Production remains off until paired regional probes and total-cost measurements justify rollout; instance retirement is a gated operational stage and is not triggered automatically by merging or setting the read share to 100%.
Publication and cleanup correctness (AI generated)
Publish immutable Workers KV values through generation-fenced, conditional R2 pointers, with Cache API in front. Parsed-memory reuse checks the publication generation. App reads preserve that revision through Cache API and parsed-memory reuse, distinguishing a pointer not yet published from a generation-bearing deletion pointer. Unpublished apps always fall back to the replica; authoritative mode cannot turn a cached publication miss into a deleted-app response. Published app tombstones and authoritative deletion pointers retain their existing behavior. Missing dependencies, delayed KV visibility, invalid documents, and oversized manifests fall back safely. Pointer lookup and JSON decoding share a 50 ms deadline; the following KV value read has its own 50 ms deadline. Sampled logs report R2 reads/time/timeouts separately from KV reads/time/kv_value_timeouts; either timeout uses the replica fallback. KV value cache TTL remains 30 seconds pending regional propagation validation.
The Vault-gated writer uses fresh, bounded primary snapshots, releases transactions before provider I/O, shares an 800-operation budget, and resumes 48-key backfill pages. Transaction-scoped enqueue deduplication prevents a 5,000-file upload from producing thousands of duplicate jobs while retaining later producer changes. Queue and conditional-write retries remain capped at five. Successful ACKs durably enqueue purges before removing work; wake failures retain pending changes. Immutable-value cleanup is bounded and resumable.
Full cleanup reuses atomically written generation metadata and the listed R2 ETag for its first conditional write, removing a redundant GET per historical key. Legacy metadata and ETag conflicts retain fresh pointer reads. Deletion fences are retained and advanced: tombstone generation 1 followed by a paused live writer at generation 2 still needs a generation-3 deletion fence. Skipping that refresh or pruning the pointer after a grace period can let generation 2 resurrect deleted state; a two-minute lease does not prove the publisher has stopped. The regression test covers that interleaving. Applying the proposed null-digest early return to deleteListed makes fences a paused generation newer than the old tombstone after cleanup fail: generation 2 publishes a live key because cleanup skipped the generation-3 deletion fence. The original code passes. Tombstone refresh retains its documented R2 PUT/list cost; skipping it is not a safe cost optimization. Normal daily reconciliation refreshes app documents without a full history scan. Initial backfill, explicit full repair and deleted-app cleanup remain proportional to historical pointer count, with storage growth and repair operations documented as rollout cost gates.
The standalone benchmark checks the local Vault sync prerequisite before creating fixtures. The producer-transaction contract test leases only its own fixture before commit, avoiding a race with parallel global claimers.
Plugin CI finishes Docker image extraction and dependency installation before starting Supabase to reduce bootstrap resource contention. The previous installer exited 137 while those phases overlapped; its kernel kill reason was not captured. The five-minute job limit and test steps remain in place.
Test Plan (AI generated)
- Current-head review: CodeRabbit approved
e58e04aaec. All 28 review threads are resolved. Its four reader/writer/SQL findings and the subsequent test-helper finding are addressed. - Current-head CI: no failed or pending checks. Backend, SQL, Cloudflare plugin db/kv-1/kv-2, unit, Playwright, lint/typecheck, CLI and published-CLI compatibility jobs passed. Skipped automation/matrix entries and a neutral check are not counted as executed tests.
- PostgreSQL contracts: all 30 passed against the migrated Supabase schema. Coverage includes internal RPC grants, durable enqueue/purge atomicity, backfill cursor preservation, app-publication reconciliation, lease/generation fencing, full-repair recovery, five-attempt archives, and real two-session claim-lock contention without spending queue attempts.
- Reader and cache regressions: missing/cached unpublished apps use replicas, deletion revisions survive cache/memory reuse, stalled Cache API/R2/KV reads fall back, missing override channels reject mismatched snapshots, public batches have bounded document fanout, and fallback happens before duplicate sink writes. DB/snippet caches stay isolated from KV responses.
- Worker integration: strict KV and DB plugin suites cover channel/override/plan behavior, real publication and fallback, historical bundles, and 5,000-file manifests. Strict mode rejects escaped SQL reads; the published CLI contract remains intact.
- Local merge validation: 351 focused tests across 20 files passed on
bc06dff696, plus backend/CLI typechecks, backend/test lint, frozen install, migration order and plugin-runtime isolation. API and plugin Worker dry-run builds passed; plugin size was 504.80 KiB / 150.88 KiB gzip. The only subsequent source change ine58e04aaecis the test helper and its contention test. - Test-helper validation: direct local checks of the actual helper passed temporary contention, empty claims, immediate SQL errors and bounded permanent contention. Full-schema CI then passed its new real contention contract. Production queue limits, runtime code and the migration were unchanged by that test-only fix.
- Independent review scope: Claude independently reviewed
b0abb8e87fand passed 163 focused units. Subsequent fixes are covered by the current-head CI and CodeRabbit approval above. Earlier Codex/CodeRabbit approvals and quota-limited attempts are not presented as fresh current-head review coverage. - Benchmark evidence: the paired local HTTP measurements, exact runtime commit, sample counts, response/provenance checks, raw data and limitations appear in the performance section and linked results document. The CPU benchmark check does not measure KV/R2 network latency.
- Quality gate: Sonar passed with zero security hotspots. Its 28 maintainability/style annotations remain; this is not a zero-issue report.
- CI startup failures were investigated before retry: Playwright shard 2 passed its targeted retry after an Edge Runtime health-check 503, and KV-1 passed its targeted retry after workerd
SQLITE_BUSY. Both failed before tests. Their underlying runner/lock causes remain unproven; no timeout, storage-layout or runtime-code change is claimed as a fix.
Production rollout has not been enabled by this work. Worldwide latency, publication freshness, fallback load and the combined Google/Cloudflare bill must pass the staged acceptance gates before regional replicas are retired.
Reproduce locally with Bun, Docker and repository dependencies installed:
bun install
bun run supabase:start
bun run supabase:db:reset
bun run supabase:with-env -- bun scripts/quiesce-test-scheduler.ts
bun run lint:backend
bun run typecheck:backend
bun run supabase:with-env -- bunx vitest run tests/plugin-kv-state-sync.test.ts
For the DB baseline, start local Workers with PLUGIN_STATE_KV=off ./scripts/start-cloudflare-workers.sh, then run bun run supabase:with-env -- bun run test:cloudflare:plugin. Stop those Workers before starting the KV run.
Use a fresh local persistence directory for strict KV validation so restored database generations and prior Cache API entries cannot affect the run:
PLUGIN_STATE_KV=on PLUGIN_STATE_KV_AUTHORITATIVE=true PLUGIN_STATE_KV_STRICT=true CLOUDFLARE_PERSIST_DIR=.context/plugin-state-pr-validation ./scripts/start-cloudflare-workers.sh
While those Workers run, in a second terminal:
bun run supabase:with-env -- bun run test:cloudflare:plugin:kv
USE_CLOUDFLARE_WORKERS=true PLUGIN_STATE_BENCH_SAMPLES=100 PLUGIN_STATE_BENCH_OUTPUT=.context/plugin-state-benchmark.json bun run supabase:with-env -- bun scripts/benchmark-plugin-state.ts
bun run check:plugin-runtime
bun run check:plugin-worker-size
Run the benchmark after tests finish. The strict KV setup enables the local Vault sync prerequisite; standalone runs on a fresh database stop with a clear prerequisite error until that flag is enabled. These commands and all reported HTTP probes are local-only.
Rollout and recovery procedure · Current benchmark, raw data and limitations
Checklist (AI generated)
- Current-head CI on
e58e04aaecpassed, including lint/typecheck and published-CLI compatibility. - This change requires documentation.
- Internal rollout, recovery and benchmark documentation is updated. No customer website documentation change is required.
- Backend E2E coverage exercises both DB and strict KV paths, including large manifests and channel transitions.
- Local Worker HTTP probes and paired benchmarks were run; reproduction commands are included above.
- This public PR contains no private customer data, unpublished advisory identifiers or embargoed exploit details.
- Security-sensitive OTA and SQL changes were reviewed against
docs/agent-security-regressions.md, including internal RPC permissions and preserved plugin/CLI contracts.
Generated with AI
Summary by CodeRabbit
- New Features
- Plugin requests can be served from synchronized state, with database fallback when state is unavailable.
- Support covers updates, channel information, statistics, and manifest-size lookups. Rollout can be controlled by environment and sampled by device.
- Bug Fixes
- KV-served responses no longer populate shared edge caches, keeping cached responses isolated from database-served traffic.
- State synchronization handles retries, deferred work, and recovery while preserving existing fallback behavior.
- Documentation
- Added design, rollout, recovery, and local benchmark documentation, including limitations and validation results.