fix(ratelimit): keep per-replica counter snapshots so replicas stop overwriting each other
Description
With several replicas on one database, every replica overwrote the same rate_limit_counters rows once a second (last writer wins), and a restarting replica restored another replica's counts.
- Each
Servicegets a random writer ID. Snapshots go to a new table/collection,rate_limit_window_snapshots, keyed additionally by writer, so replicas never overwrite each other. - Restore rolls each saved row forward to the current window and takes the max per bucket (current and previous, requests and tokens), never the sum. A single instance restarting under a new ID gets its own counts back. With N replicas it approximates one replica's share.
- A reset applies to every replica. The store gives the rule (every rule, for reset-all) a new counter epoch and deletes every writer's rows in one write, and the change is announced through config sync, so each replica's reload clears its counters within a few seconds. Rows saved under the old epoch in the meantime, by another replica's flush or by a legacy import racing the reset on standalone MongoDB, are never restored. Before this, a reset only cleared the replica that served it.
- Each rule has a counter epoch, drawn by the store when the rule is created, switches between shared and per-child (in the same write), or is reset, and kept on limit edits. Saved rows carry the epoch they were counted under and restore skips any other epoch, so a mode change or delete-and-re-add never restores cleared counts — without depending on a row delete succeeding, surviving a restart, or touching rows other replicas saved under the new rule.
- Migration: existing rows are imported when the store opens and the legacy table is emptied, not dropped (MongoDB deletes the imported documents in batches of 10,000). Older binaries keep working on the legacy table during a rolling upgrade, and a rollback is clean. An in-place primary-key change would have broken older binaries' flushes, which is why this uses a new table.
- Docs: persistence paragraph and reset section in
features/rate-limits.mdx, including a note to finish a rolling upgrade before resetting or deleting rules; the replica table and config-sync scope inguides/production.mdx.
Admission is still per replica (about N× the configured limit). This PR makes the saved counts and resets correct across replicas; shared enforcement is a follow-up.
Tests: two writers keep separate rows, max-merge restore, restart with a new writer ID, reset clears all writers, a reset on one replica clears another and rows saved under the old epoch are not restored, the refresh holds the persist lock between the rule swap and the counter reset (a hook, no sleep), reset draws a new epoch on every backend, a legacy count written back after a reset is not restored on MongoDB, counter epochs (mode round trip with failing deletes, rule changed while stopped, rows saved under the new rule kept, limit edits keep counts, pre-epoch rules keep their counts), and legacy import on SQLite, PostgreSQL and MongoDB.
Summary by CodeRabbit
-
Bug Fixes
- Rate-limit counters are saved separately by each replica, preventing one replica from overwriting another’s saved counters. After a restart, restored limits use the highest saved count per time window rather than summing counts across replicas.
- Resetting a rate limit clears its saved records across replicas. Other running replicas retain their in-memory counters and may save them again later.
- Existing saved counters are carried forward during upgrades. Changing a rule’s per-child setting, or deleting and recreating a rule, starts its counts at zero; editing a limit preserves counts.
-
Documentation
- Clarified how rate-limit counters are saved, restored, and affected by resets across replicas.