fix(engine): page message history in numeric snowflake order

WI
1d ago
+10200-86

Problem

Message ids are snowflakes stored as decimal TEXT (packages/engine/src/engine/snowflake.ts: epoch 2025-01-01, timestamp << 22). They widened from 17 to 18 digits around 2025-10-03. Three history queries ordered by messages.id DESC and paged with lt(id, before) / gt(id, after), which compare as text. History that spans the width change came back misordered: every 9… 17-digit id sorted newer than every 1…/2… 18-digit id. Cursors also skipped across the boundary; for example, before=<oldest 18-digit id> returned nothing.

  • getMessages (engine/message.ts): GET /v1/channels/:name/messages
  • getDmMessages (engine/dm.ts): GET /v1/dm/:conversation_id/messages. Its comment claimed ids were "fixed-width (19 digits)"; that comment is removed.
  • getDmMessagesForWorkspace (engine/dmAll.ts): workspace-key GET /v1/dm/conversations/:conversation_id/messages

CodeRabbit flagged this on #698. The bug predates that PR and is fixed here separately because the fix needs an index migration.

Fix

  • retention.ts gains beforeSnowflake and snowflakeDesc next to afterSnowflake. They compare and order by (length(id), id). All three take an optional handle; given one, Postgres compares id COLLATE "C" (new sqlx.binaryText) to match the idCollateC index expressions. Existing afterSnowflake callers are unchanged.
  • New engine/snowflakePage.ts newestSnowflakePage() is shared by the three queries. SQLite can't seek a row-value bound (length(id), id) < (L, c) on an expression index; I checked with EXPLAIN QUERY PLAN. A single-statement before page would therefore read every newer row of the same width first, and D1 bills every row read. So a before page does at most two index seeks:
    1. length(id) = L AND id < before ORDER BY id DESC, a range seek within the cursor's width;
    2. only if that page is short, length(id) < L ORDER BY length(id) DESC, id DESC for the remainder.
      Pages without a cursor, and after pages, are a single seek, ordered by length(id) DESC, id DESC.
  • Channel listing pins INDEXED BY idx_messages_channel_length_id, following the thread.ts precedent. Its workspace_id filter could otherwise let SQLite pick the workspace index and sort.
  • Ordering of equal-width ids is unchanged.

Index: replaces idx_messages_channel_time

New idx_messages_channel_length_id on messages(channel_id, length(id), id). On Postgres the id is indexed COLLATE "C".

  • SQLite: 0087_messages_channel_length_id_index.sql creates it, then drops the old index.
  • Postgres: 0008_messages_channel_length_id.sql is generated by generate:pg-migrations, with its snapshot and journal entry. Both statements run in one transaction.
  • schema.ts and schema.pg.ts are updated, and migrations.generated.ts is regenerated with generate:actor-migrations.

Replaces rather than sits alongside it: every channel_id lookup the old (channel_id, id) index served is a prefix of the new one. After this change, nothing orders a channel by bare text id. Keeping both would only add write cost to every message insert.

Tests

New conformance/mixedWidthHistory.test.ts. A channel and a DM conversation each hold three 17-digit and three 18-digit ids; the DM also holds its real, current opener message. The test covers the channel route, the agent DM route and the workspace-key DM route. It asserts:

  • the full newest-first order;
  • before across the boundary, both entirely and mid-page;
  • after across it, and after plus before together;
  • unchanged equal-width cursors;
  • that walking limit=2 pages visits every message exactly once, in order;
  • on SQLite, that both page shapes seek idx_messages_channel_length_id without a temp B-tree.

The test fails on main, with 17-digit ids listed first, and passes here. Four migration-list tests are updated for the new files.

  • npm test (engine): all passing
  • npm run lint, npm run typecheck (engine): clean
  • npm run check:pg-migrations: in sync; scripts/check-published-engine-migrations.test.mjs: passing
  • Not run locally: test:conformance:pg, because no Postgres fixture is available here. CI's PG conformance job covers the new test in PG mode.

Follow-ups (not in this PR)

The same text ordering remains in activity.ts (workspace activity feed, desc(messages.id)) and thread.ts (thread replies, id ASC). Fixing those needs (workspace_id, …) / (thread_id, length(id), id) indexes of their own.

🤖 Generated with Claude Code


Run Proof
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.