ci: wait for dockerd before launching the test-service coordinator on Linux agents

Closed
RO
3mo ago
+40-1

What

test/js/sql/sql-mysql-query-string-leak.test.ts and test/regression/issue/26030.test.ts went red on :alpine: 3.23 x64 shard 0 in build 74556 (PR #34477, unrelated to its diff):

failed to connect to the docker API at unix:///var/run/docker.sock; ... dial unix /var/run/docker.sock: connect: no such file or directory
error: A functional `docker` is required in CI for some tests.
      at isDockerEnabled (test/harness.ts:1056:17)

error: Failed to start service mysql_plain (via coordinator): Docker is not available. Please ensure Docker is installed and running.

All four in-shard retries hit the same state because the docker daemon on that VM never came up for the life of the job.

Why

scripts/agent.mjs installs the buildkite-agent service with no ordering against docker:

# openrc                      # systemd
depend() {                    After=syslog.target
  need net                    After=network-online.target
  use dns logger
}

so on a cold ephemeral boot the agent can accept a job before dockerd is listening. Grepping the shard logs for that build: on every Alpine shard the runner's --- Docker section (docker ps) prints "failed to connect to /var/run/docker.sock", and the coordinator's one-shot docker version a few seconds later normally wins the race (shards 1-19 and the x64-baseline twin of shard 0 all went on to coordinator: mysql_plain ready). On this one VM it lost, and docker version was still failing 3+ minutes later when the test file ran, so the daemon never came up at all on that boot.

On Debian/Ubuntu the same --- Docker section already shows a healthy CONTAINER ID ... header at runner start, so this is specific to the openrc boot ordering.

Fix

  • scripts/runner.node.mjs: on CI Linux, if the docker CLI is present but docker version can't reach the daemon, poll it for up to 30s before spawning the test-service coordinator. This closes the cold-boot race for the current images at zero cost on hosts where the daemon is already up. If the daemon is still unreachable after 30s the shard is on a VM where dockerd failed to start; the runner annotates and exits 75 (EX_TEMPFAIL) so Buildkite's automatic-retry (limit: 1, Linux test agents are ephemeral acquire-job/disconnect-after-job) re-runs the shard on a fresh VM instead of burning 4x retries of every container-backed test against a dead host. A fleet-wide docker regression still surfaces on the second attempt.
  • .buildkite/ci.mjs: add exit_status: 75 to getRetry()'s automatic list alongside the existing -1/255 infra-retry codes.
  • scripts/agent.mjs: declare after docker / use ... docker in the openrc service and After=docker.service in the systemd unit so newly baked images boot with the correct ordering and the wait loop above is not doing load-bearing work.

Verification

This is CI machinery with no src/ or test changes; the reported test already passes on main HEAD whenever docker is reachable (299/300 recent builds). The fix cannot be demonstrated as fail-before/pass-after locally: without a reachable docker daemon the test is skipped (isDockerEnabled() returns false outside CI), and with one it already passed.

What I did verify:

  • node --check passes on all three modified files.
  • The new branch runs inside the existing isCI && isLinux && docker compose version guard, so non-Linux / dockerless hosts are untouched and a healthy daemon takes the fast path (one extra spawnSync docker version).
  • Sampled five Alpine shard logs from build 74556: all show docker ps failing at runner start and docker becoming ready within the 30s budget, so the poll is sized against real numbers rather than guessed.
  • openrc use/after and systemd After= are ordering-only (not need/Requires=), so images without docker still boot the agent.

no test proof · iteration 0 · build/CI scripts only; test-proof not applicable