ci: serialize docker service startup through a per-shard coordinator

Merged
RO
5mo ago
+301-83

Problem

Docker service tests flake in CI with:

Failed to start service postgres_plain: Error: Failed to start service postgres_plain:  Container bun-test-services-postgres_plain-1  Creating
 Container bun-test-services-postgres_plain-1  Error response from daemon: Conflict. The container name "/bun-test-services-postgres_plain-1" is already in use by container "479c2174acab...". You have to remove (or rename) that container to be able to reuse that name.

Example: https://buildkite.com/bun/bun/builds/61534#019eae6b-df63-4bed-af93-708aa3b16273 (sql.test.ts, passed only after a runner retry).

Cause

docker compose up is not safe to run concurrently for one project: two invocations can both observe the container as missing and race to create it, and the loser exits with the name conflict above. CI ran exactly that race: scripts/runner.node.mjs backgrounded test/docker/warmup-ci.ts (one compose up -d for all services the shard needs, not awaited) while the first docker-using test's ensure() ran its own compose up -d --wait for the same service. The ps -a output in the failure shows the other invocation's container in Created state, less than a second old.

Fix

Make exactly one process own compose, instead of retrying losers. The runner now spawns test/docker/coordinator.ts once per shard:

  • The coordinator listens on a unix socket and is the only process that runs docker compose. Concurrent requests for one service collapse onto a single compose up --wait via the existing in-flight promise map, so the create race cannot happen.
  • ensure() in test/docker/index.ts becomes a client when BUN_DOCKER_COORDINATOR is set (the runner exports it after the coordinator prints its ready line): it sends the service name and waits for the coordinator's reply with the port mapping. An ok=false reply is a real service failure and is thrown, not retried.
  • The coordinator subsumes warmup-ci.ts (deleted): the same path-to-service map prestarts the services the shard's tests need at launch, in the background.
  • Without the env var (local dev, or the coordinator is unreachable), ensure() falls back to running compose directly, same as before.
  • Coordinator lifetime is tied to the runner through its stdin pipe: EOF means the runner is gone and the coordinator exits.

Verification

The signal for this infrastructure is the docker-backed suites (sql, valkey, s3, autobahn, squid) passing through the coordinator on the Linux CI lanes, which they did on the first coordinator build (61566); no dedicated test file is included, per review.

Also smoke-tested the runner handshake end to end under Node: spawn coordinator, wait for COORDINATOR_READY, a client with a poisoned docker on PATH gets service info through the socket only (proving it never shells out to compose), and the coordinator exits on stdin EOF when the runner goes away.