perf(worker): parallelize per-fingerprint issue upserts in telemetry ingest #SUPERLOG

Closed
SU
4d ago
+83-19

Summary

Every telemetry ingest batch is running 9–10 seconds (70% above the 5,618 ms baseline), causing a sustained 13–39 second cursor lag for all tenants. Fingerprinting and issue creation are delayed for every project on the shared worker.

Root cause

flushIssueGroups in apps/worker/src/telemetry/ingest.ts iterates all distinct fingerprint groups with a sequential for…await loop, making one blocking Postgres round-trip per group. Each upsertIssue call runs a multi-CTE SQL statement (~50–100 ms). With 100–200 distinct fingerprints per BATCH_SIZE=500 batch — driven by project f0686143 generating many unique not-found span fingerprints — the serial round-trips alone consume 5–20 seconds of wall-clock time per batch.

ClickHouse P95 (2,049 ms, improved vs baseline) and LLM concurrency (down 80%) are both confirmed not to be the bottleneck. The pattern holds since the current worker boot at 11:17 UTC: lifetime span average = 9,352 ms (sum 12,962,042 ms / count 1,386 batches), log average = 9,843 ms.

The upserts are safe to parallelize: each targets a distinct (project_id, fingerprint) primary key so there are no intra-batch row conflicts.

Remediation

Replace the sequential for…await loop in flushIssueGroups with a bounded concurrent map (UPSERT_CONCURRENCY=8), using the same mapWithConcurrency pattern already established in agent-runs/prompt-context.ts.

  • Phase 1: run all upserts concurrently up to 8 at a time (leaving headroom in the postgres.js default pool of 10 for the tick's own queries and pg-boss workers).
  • Phase 2: dispatch transitions for new/recurred issues sequentially — these pg-boss INSERTs are sparse and fast.

Error isolation is preserved: each upsert failure is caught, logged, and counted independently without affecting other groups or the cursor advancement.

Testing

All 11 existing telemetry-ingest unit tests pass, including the NUL-byte isolation test and the fingerprint-collapse test.

Related incident: 9e740fa3-4cdc-4a88-a456-fc2e00644e7e

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

Summary by cubic

Parallelizes the per-fingerprint issue upserts in telemetry ingest so batches stop blocking on sequential Postgres round-trips. This cuts per-batch processing from ~9–10 seconds and removes the sustained 13–39 second cursor lag affecting all tenants.

  • Upserts run concurrently 8 at a time via a bounded mapWithConcurrency helper (same pattern as agent-runs/prompt-context.ts), leaving headroom in the postgres.js default pool of 10.
  • Parallelism is safe: each upsert targets a distinct (project_id, fingerprint) key, so there are no intra-batch row conflicts.
  • Transition dispatches for new/recurred issues still run sequentially, and per-group error isolation is preserved—failures are logged and counted without wedging the cursor.