fix(sync): stop activity writes and polls from locking the database on slow storage
Sync requests sometimes failed in bursts with 503 "the database is busy" responses, and some requests got no response at all. Successful writes in those bursts took 40 to 77 seconds.
Activity events in sync_event carry padded payloads of tens of kilobytes. SQLite keeps most of each row in a chain of overflow pages, and created_at and expires_at come after the payload, so reading either column walks the whole chain. Three queries filtered on expires_at:
sync_event::appenddeletes the account's expired events in every write transaction that carries activity, while it holds the write lock.event_idsruns on every/changespoll, and twice when the poll waits.listfor/eventsloaded every payload and only then dropped the already processed broadcasts in Rust.
For an account with 100 stored events, each of these queries read about 2,150 pages, or 8.6 MB. On slow storage the write lock stayed held long enough for waiting writers to hit the 30 second busy timeout. Those writers also kept their pooled connections, so read-only requests waited for a connection too.
A new migration adds the index sync_event_account_expiry on (account_id, recipient, expires_at, id) for SQLite and PostgreSQL. It takes the three queries to about 6 pages. list now filters broadcasts by since in SQL and returns the same events as before. Building the index reads every stored event once, so the migration takes a while on large databases.
The old /events query also sorted every matching payload. For an account with 100 events that sort outgrew SQLite's page cache and spilled about 7 MB to a temporary file on every request. With the filter in SQL, a poll with nothing new sorts nothing.
When the connection pool timed out, get_db_conn! panicked and the client saw a dropped connection. It now returns the same retryable 503 with Retry-After: 30 as a locked database does.
The metadata validation tests fetched live YouTube feeds, which now fail intermittently with 404 and 500 responses and failed CI on main too. They now use local synthetic feeds, so the test suite needs no network.
Proof: I reproduced the failure with the release build, synthetic data and 5 ms of added latency per disk read. 77 requests failed, 6 as busy responses and 71 as pool timeouts, and writes took up to 50 seconds. The same run without stored events had no failures, and so did the run with this change, where writes stayed under 6 seconds. writes_and_polls_do_not_read_stored_event_payloads breaks the stored events' overflow chains on disk and fails on the old code with "database disk image is malformed". an_exhausted_pool_is_reported_as_retryable fails on the old code with the panic. cargo test (also without network access), cargo fmt --check and cargo clippy --all-targets pass, clippy also with --no-default-features --features postgres. I applied and reverted the PostgreSQL migration on PostgreSQL 17.
Reviewed with sol-loop: 1 round with gpt-6.1-sol on xhigh.
Sol verdict: approved
Changes made by Claude Opus 5.5 in Claude Code (T3 Code).