fix(batch): let I/O-bound seal repair run 48 workers

Merged
GI
8h ago
+30-8

Problem

Seal repair is the record_pending bottleneck: ~5.4k rows and growing ~1,156/h (lead2 ruling 072b5e). Pass 29898470 selected 1,024 rows but attempted only ~157 before the 1,500 s run deadline. The other 867 count as deferred, which means not attempted, not refused (coordinate() in record_seal_cluster_repair.py). That is 16 workers x 1,500 s at ~150 s per row.

The pod is not CPU- or memory-bound (in-cluster Prometheus, 10 min): CPU ~0.7 of the 4-CPU limit, CFS throttling 0, working set ~1.1 GB of 8 Gi. Rows wait on I/O, so throughput scales with workers. Both config/record_seal_repair.py and the chart template capped workers at 16, and prod already ran 16. More rows per pass or a shorter interval would not help: the pass is time-bound, and Forbid plus a 1,500 s run already runs back to back.

Fix

  • Cap workers at MAX_WORKERS = 64 (was 16), in code and in the chart guard.
  • Prod recordSealRepair.workers: 16 -> 48.
  • Repair pod requests 1 CPU / 2 Gi -> 2 CPU / 4 Gi; limits unchanged (4 CPU / 8 Gi). At ~70 MB per worker, 48 workers is ~3.3 GB.

Expected: ~3x attempted rows per pass (~470), ~900-1,000 sealed/h against ~1,156/h new. It ships on the next batch-pipeline-worker image/chart roll, which also activates #24703's jobs leg (not live today: CronJob image ced945db predates it).

Tests

BACKEND_UNIT_TEST_FILE_LIST=... bash test.sh over test_record_seal_cluster_repair.py (+7 new cases: 1/16/48/64 accepted; 0/65/128 refused), test_record_seal_cluster_repair_deploy.py (prod renders 48; 65 refused), test_record_seal_sweeper.py and test_record_seal_worker_guards.py: 77 + 28 + 13 + 8 passed.

Failure-Class: none

(Capacity tuning: a worker cap sized before the record_pending surge; not a recurrence of a defect class.)

Risk / rollback

Firestore/GCS read concurrency from repair rises ~3x (each row does the same proof reads as today). Watch the seal-repair summary (sealed, deferred, budget_exhausted), pod memory, and Firestore contention. Rollback: prod recordSealRepair.workers back to 16 (values only).

Evidence: omi-workspace data/run-2026-11-05-omi-pipeline/seal-repair-throughput-2030Z.md.

šŸ¤– Generated with Claude Code