fix(engine): page message history in numeric snowflake order
Problem
Message ids are snowflakes stored as decimal TEXT (packages/engine/src/engine/snowflake.ts: epoch 2025-01-01, timestamp << 22). They widened from 17 to 18 digits around 2025-10-03. Three history queries ordered by messages.id DESC and paged with lt(id, before) / gt(id, after), which compare as text. History that spans the width change came back misordered: every 9… 17-digit id sorted newer than every 1…/2… 18-digit id. Cursors also skipped across the boundary; for example, before=<oldest 18-digit id> returned nothing.
getMessages(engine/message.ts):GET /v1/channels/:name/messagesgetDmMessages(engine/dm.ts):GET /v1/dm/:conversation_id/messages. Its comment claimed ids were "fixed-width (19 digits)"; that comment is removed.getDmMessagesForWorkspace(engine/dmAll.ts): workspace-keyGET /v1/dm/conversations/:conversation_id/messages
CodeRabbit flagged this on #698. The bug predates that PR and is fixed here separately because the fix needs an index migration.
Fix
retention.tsgainsbeforeSnowflakeandsnowflakeDescnext toafterSnowflake. They compare and order by(length(id), id). All three take an optional handle; given one, Postgres comparesid COLLATE "C"(newsqlx.binaryText) to match theidCollateCindex expressions. ExistingafterSnowflakecallers are unchanged.- New
engine/snowflakePage.tsnewestSnowflakePage()is shared by the three queries. SQLite can't seek a row-value bound(length(id), id) < (L, c)on an expression index; I checked withEXPLAIN QUERY PLAN. A single-statementbeforepage would therefore read every newer row of the same width first, and D1 bills every row read. So abeforepage does at most two index seeks:length(id) = L AND id < before ORDER BY id DESC, a range seek within the cursor's width;- only if that page is short,
length(id) < L ORDER BY length(id) DESC, id DESCfor the remainder.
Pages without a cursor, andafterpages, are a single seek, ordered bylength(id) DESC, id DESC.
- Channel listing pins
INDEXED BY idx_messages_channel_length_id, following thethread.tsprecedent. Itsworkspace_idfilter could otherwise let SQLite pick the workspace index and sort. - Ordering of equal-width ids is unchanged.
Index: replaces idx_messages_channel_time
New idx_messages_channel_length_id on messages(channel_id, length(id), id). On Postgres the id is indexed COLLATE "C".
- SQLite:
0087_messages_channel_length_id_index.sqlcreates it, then drops the old index. - Postgres:
0008_messages_channel_length_id.sqlis generated bygenerate:pg-migrations, with its snapshot and journal entry. Both statements run in one transaction. schema.tsandschema.pg.tsare updated, andmigrations.generated.tsis regenerated withgenerate:actor-migrations.
Replaces rather than sits alongside it: every channel_id lookup the old (channel_id, id) index served is a prefix of the new one. After this change, nothing orders a channel by bare text id. Keeping both would only add write cost to every message insert.
Tests
New conformance/mixedWidthHistory.test.ts. A channel and a DM conversation each hold three 17-digit and three 18-digit ids; the DM also holds its real, current opener message. The test covers the channel route, the agent DM route and the workspace-key DM route. It asserts:
- the full newest-first order;
beforeacross the boundary, both entirely and mid-page;afteracross it, andafterplusbeforetogether;- unchanged equal-width cursors;
- that walking
limit=2pages visits every message exactly once, in order; - on SQLite, that both page shapes seek
idx_messages_channel_length_idwithout a temp B-tree.
The test fails on main, with 17-digit ids listed first, and passes here. Four migration-list tests are updated for the new files.
npm test(engine): all passingnpm run lint,npm run typecheck(engine): cleannpm run check:pg-migrations: in sync;scripts/check-published-engine-migrations.test.mjs: passing- Not run locally:
test:conformance:pg, because no Postgres fixture is available here. CI's PG conformance job covers the new test in PG mode.
Follow-ups (not in this PR)
The same text ordering remains in activity.ts (workspace activity feed, desc(messages.id)) and thread.ts (thread replies, id ASC). Fixing those needs (workspace_id, …) / (thread_id, length(id), id) indexes of their own.
🤖 Generated with Claude Code

Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.