perf: one compile for identical jobs, output modes, configurable pools and latency alerts

Merged
SH
10d ago
+477-32

Stacked on #134 (this branch contains its commits; review only the last commit).

One compile for the same source

The compile cache was a plain map with no request coalescing, so jobs with the same source that arrived together each started their own compile and the cache only helped the ones that came after. The Platform's judge sends one job per test, so N tests of one submission meant N simultaneous compiles of the same file.

Now the first job compiles and the others wait for it (a waiter gives up if its own context ends):

  • success: every waiter uses the compiled binary, and later jobs hit the cache;
  • a source that does not compile: the waiters get the same compile error, and nothing is cached;
  • a sandbox error in the compiling job: each waiter tries again for itself, so one job's transient failure is not copied to the others;
  • cache TTL 0 (disabled): every job compiles, as before.

output mode

POST /api/v1/execute accepts output: full (default, unchanged), preview (256-byte previews only) or none. Full stdout and stderr, up to 8 MB per test, otherwise go through Redis, the event log, every SSE client and the webhook. Stored with the job; absent on older jobs, which mean full.

Pools and alerts

  • AC_DB_MAX_CONNS, AC_REDIS_POOL_SIZE, AC_REDIS_MIN_IDLE_CONNS (defaults 128, 256, 32, as before), validated. Ingest can use 512 database connections at once, so the README says to raise the pool first under heavy submission traffic.
  • Prometheus rules for queue-wait p99 over 30 s, claim/load/persist/publish p99 over 1 s, and a compile cache that almost never hits.

Considered and dropped

Overlapping the status write with its event publish (COMPILING, RUNNING). It saves a sub-millisecond Redis round trip and could publish an event for a job whose verdict another worker has just written; not worth it.

Testing

go vet clean; go test -race ./... passes except internal/db (needs Docker; fails the same way on development). New tests: shared compile across 20 concurrent callers, shared compile failure, transient-error retry, a waiter's context, TTL 0, output modes, queue round trip, gateway rejection of a bad mode, pool-size validation. The alert expressions were not run against Prometheus (promtool is not installed here).

Summary by cubic

Shares one compile across jobs with the same source, language, files, and flags that arrive together, so the Platform's judge (one job per test of the same submission) no longer starts N simultaneous compiles of the same file. Also adds an output mode to POST /api/v1/execute so callers that only need verdicts can keep bulk stdout and stderr out of Redis, the event log, SSE, and webhooks, makes database and Redis pool sizes configurable, and adds Prometheus latency alerts.

  • The first job compiles; waiters reuse the result, and later jobs hit the cache. A source that fails to compile fails the same way for all waiters and is not cached.
  • If the compiling job hits a transient sandbox error, each waiter retries for itself; with a cache TTL of 0, every job compiles as before.
  • output accepts full (default), preview (256-byte previews only), or none; jobs enqueued before this change lack the field and behave as full.
  • AC_DB_MAX_CONNS (default 128), AC_REDIS_POOL_SIZE (256), and AC_REDIS_MIN_IDLE_CONNS (32) are validated; the database connect path also rejects a pool size outside 4–2000, so a bad value fails at startup. The README advises raising the DB pool first because ingest can use 512 connections at once.
  • Alerts fire on queue-wait p99 over 30 s, claim/load/persist/publish p99 over 1 s, and a compile cache miss rate above 0.9.