perf: stage timings, shared event subscription and honest Retry-After

Merged
SH
10d ago
+615-36

Latency work for the p95/p99, built on top of #132 (this branch includes its commit; merge that first, or review only the last commit).

See where the time goes

  • ac_queue_wait_seconds{stream}: enqueue to a worker reading the job, taken from the Redis stream id's timestamp, so no new field on the message.
  • ac_job_stage_seconds{stage}: claim, load_tests, run, persist, publish, beside the existing compile histogram. README gains a "Performance Tuning" section with the PromQL for each and the sizing rules.

Less waiting per job

  • A worker now fetches the job's tests from Redis while it claims the job in PostgreSQL, instead of one after the other.
  • Java and Kotlin start with the serial collector and shared class data (-XX:+UseSerialGC -Xshare:auto; javac gets the same through -J), which also lowers each run's thread count.

Fewer connections, truthful overload

  • One Redis pattern subscription per gateway process serves every SSE client (it was one per client; 5,000 at the benchmark's extreme tier). The subscription is confirmed before the first client registers, so subscribe-before-replay still holds. A client that falls behind has its channel closed, which the SSE handler already turns into a reconnect that replays from the event log, so nothing is dropped silently. PubSub.Wait stops the subscription.
  • When the queue is full, 429 carries Retry-After equal to the time the queue needs to fall under the admission limit at the measured completion rate (1 to 60 s; the old fixed 10 s when no rate is known yet).

Not done, and why

  • A pre-warmed Go build cache: go-judge needs a writable cache directory inside each sandbox, and I could not try mounts without Docker. Worth an experiment on the real host.
  • Running one engine and go-judge pair per NUMA node, and raising go-judge's -pre-fork: documented in the README, but they are deployment changes to measure on the 256-thread host.
  • Batching database writes: results and verdict are already written in one transaction.

Testing

go vet clean; go test -race ./... passes for every package except internal/db (needs Docker, fails the same way on development). New tests cover queue-wait parsing, the drain-rate tracker, the hub (shared subscription, isolation, cancellation, slow client, stop) and the Retry-After computation. No load run: the gain is unmeasured.

Summary by cubic

Latency work for p95/p99, built on top of #132 (merge that first, or review only the last commit). Adds queue-wait and per-stage histograms, cuts Redis connections from one subscription per SSE client to one per gateway, overlaps the database claim with the Redis test load, and makes Retry-After reflect the measured drain rate.

Observability

  • ac_queue_wait_seconds{stream} derives the wait from the Redis stream ID timestamp, so no extra field on the message.
  • ac_job_stage_seconds{stage} covers claim, load_tests, run, persist, and publish; the README gains a Performance Tuning section with PromQL and sizing rules.

Less waiting, fewer connections

  • A worker fetches a job's tests from Redis while it claims the job in PostgreSQL, instead of one after the other.
  • One Redis pattern subscription per gateway process serves every SSE client; a client that falls behind is closed so it reconnects and replays from the event log.
  • 429 now sends Retry-After equal to the time the queue needs to fall under the admission limit at the measured completion rate (1 to 60 s; the old fixed 10 s when no rate is known).
  • java and kotlin start with the serial collector and shared class data, which lowers each run's thread count.

Summary by CodeRabbit

  • New Features
    • Added queue wait-time and job-stage metrics, with guidance for using them to identify performance bottlenecks.
    • Queue-full responses now include a Retry-After estimate based on observed processing rates.
  • Improvements
    • Job event streaming now uses a shared connection and can recover clients that fall behind.
    • Java and Kotlin execution settings have been adjusted.
    • Test loading can run alongside job claiming to reduce processing delays.