Raise default ingest consumer concurrency to 8 loops per process #SUPERLOG

Closed
SU
7d ago
+15-2

Summary

The ingest queue message.age_ms histogram averaged 212 s (~3.5 min) during the Oct 9 incident, a 3.16× spike over the 67 s baseline. Delivery duration rose only 5%, ruling out ClickHouse slowness — the extra latency was pure queue-wait time caused by the consumer not draining fast enough.

Root cause

Each consume loop follows a blocking batch pattern: it calls ReceiveMessage(MaxNumberOfMessages=10), awaits Promise.all(processMessage × 10), then deletes and loops. This means throughput per loop is bounded by batchSize / per-batch-insert-time = 10 / 2.5 s = 4 msg/s. With the previous default of 4 concurrent loops per process, one process caps at 16 msg/s, and any task disruption (rolling deploy, OOM kill) reduces capacity proportionally with no headroom.

The ClickHouseIngestWriter coalesces all concurrent insert() calls within a batchLingerMs window into one ClickHouse INSERT, so raising the loop count from 4 to 8 doesn't increase the number of ClickHouse inserts — it doubles how many SQS messages are coalesced into each insert, doubling per-task throughput (80 messages / 2.5 s = 32 msg/s per process).

Fix

  • Raise DEFAULT_CONSUMER_CONCURRENCY from 4 to 8.
  • Update .env.example to match.
  • Add a test asserting the new default is applied when INGEST_QUEUE_CONSUMER_CONCURRENCY is not set (the existing bounds test at ≥ 32 → 32 is unchanged).

Deployment-side environments that set INGEST_QUEUE_CONSUMER_CONCURRENCY explicitly are unaffected by this default change.

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

Summary by cubic

Raises the default ingest consumer concurrency from 4 to 8 SQS receive loops per process so the queue drains fast enough under load. During the Oct 9 incident, ingest queue message age averaged 3.5 minutes (3.16× the 67 s baseline) while delivery duration rose only 5%, showing the bottleneck was queue-wait time, not ClickHouse.

ClickHouseIngestWriter already coalesces concurrent inserts within a batchLingerMs window, so doubling the loops doubles the messages coalesced per insert rather than adding ClickHouse insert overhead. Deployments that set INGEST_QUEUE_CONSUMER_CONCURRENCY explicitly keep their current value; the change also updates .env.example and adds a test asserting the new default.