#44041 ci: build the MinIO test image from source and fail the bake on a failed image pull

Where it fails

• Main: Buildkite builds 120892 (8d7e061507) and 120898 (36cd1514ec), lanes debian 13 x64, ubuntu 25.04 x64, alpine 3.23 x64. Build 120876 (37da174d50) passed. The builds of a PR also run debian 13 x64-asan, and the test fails there too (build 120884).
• #43882 changed scripts/agent.ts, which renames every machine image. Build 120835 baked the Linux images. In each of the six bake logs the pull step takes about 2 s and prints no warning. The build step takes 90 s to 141 s.
• The Linux aarch64 lanes do not show the failure: isDockerEnabled() (test/harness.ts:1137) returns false there.
• s3.test.ts is the only test that uses the minio service.

What is gone (anonymous pull token, manifest of the tag, 2026-09-26)

minio/minio on GitHub is archived. Its README says that the community edition is distributed as source only. The release of 2025-10-15 has no binaries.

The MinIO build

• go install -trimpath -ldflags "-s -w" github.com/minio/minio@v0.0.0-20251015172955-9e49d5e7a648 with Go 1.24.13 and CGO_ENABLED=0, for linux/amd64 and linux/arm64. Both compile. MinIO's go.mod asks for toolchain go1.24.8.
• On 2 CPUs with the modules already downloaded: 154 s, peak memory 1.6 GB. A Linux bake machine is a t3.large or a t4g.large (2 CPUs, 8 GB).
• The module cache is 1.1 GB and the build cache is 0.9 GB. The Dockerfile removes both in the layer that creates them.
• Docker data per Linux machine image, from the layer sizes: 275 MB for the Go image, 9 MB for Alpine, and the 111 MB binary in the build layer and in the image.
• The base images are pinned by digest. BuildKit resolves a tag at the registry on each build, so a tag that moves misses the cache on the test machines. It resolves a digest from the local store.

The tests in CI (build 120924, machine images of this PR)

• s3.test.ts on debian 13 x64, debian 13 x64-asan, ubuntu 25.04 x64 and alpine 3.23 x64: 431 pass, 150 skip, 0 fail. Build 120876 on main, the last one that passed, has the same counts.

• The test machines reuse the layers of the bake. They do not compile MinIO. Time from coordinator: ensuring minio to coordinator: minio ready:

  Lane	main, build 120876 (pulled image)	this PR (built image)
  debian 13 x64	22.6 s	17.5 s
  ubuntu 25.04 x64	20.2 s	19.9 s
  alpine 3.23 x64	64.7 s	65.2 s
  debian 13 x64-asan	main does not run this lane	32.4 s

• Build 120947 (0f3f97a933) ran on the same machine images without a bake. All 181 jobs passed, s3.test.ts has the same counts on the four lanes, and the service is ready after 18.6 s to 66.2 s.

• test/internal/docker-prepare-ci.test.ts takes 51 ms to 65 ms on the release lanes and 300 ms on x64-asan. With the debug build one run of prepare-ci.ts takes 2.8 s to 5 s, so each test has a timeout of 30 s.

The tests without Docker (before the first bake)

• I started the linux/amd64 binary with the command and the environment of the image (Alpine's busybox sh, /data, ports 9000 and 9001) and ran s3.test.ts with BUN_TEST_SERVICE_minio=127.0.0.1. The debug build and the canary give 306 pass, 270 skip (R2), 1 fail.
• The fail is MinIO > s3 > credentials > should error with invalid endpoint. It expects PermanentRedirect from s3.us-west-1.amazonaws.com and gets ProxyConnectFailed from the proxy of my machine.
• The official binary of RELEASE.2025-09-07T16-13-09Z gives the same result for each test.
• The health check is wget from Alpine 3.23.6's busybox on /minio/health/cluster, the endpoint that mc ready asks. It exits 0 when the server is ready and 1 for a refused connection or an HTTP error.
• The old test made the bucket with mc mb data/buntest, which makes the directory /data/buntest. The image makes the same directory before the server starts.

The bake

• Image names on main: linux-x64-debian-c39a29352fc7a74c, linux-x64-ubuntu-3cc9581912aa860a, linux-x64-alpine-a621ba5e47c50ed6. With this PR: linux-x64-debian-bb6b1a1212c0ad21, linux-x64-ubuntu-2a61d6228c2418f7, linux-x64-alpine-16410fec122531c1. The aarch64 and Windows images get new names too.

• docker compose pull without --ignore-pull-failures also fails for each service with a build: section, because bun-postgres-plain:local is in no registry. --ignore-buildable leaves those services out. Only redis_plain has no build: section now.

• docker compose build builds all services at the same time. The database Dockerfiles give their server 180 s to start.

• The Docker build step of each bake, build 120835 (before) and build 120924 (this PR):

  Image	Before	This PR
  debian 13 x64	103 s	538 s
  ubuntu 25.04 x64	112 s	517 s
  alpine 3.23 x64	141 s	523 s
  debian 13 aarch64	90 s	369 s
  ubuntu 25.04 aarch64	90 s	398 s
  alpine 3.23 aarch64	118 s	429 s

Other designs

• A fork of MinIO under oven-sh that publishes the image (as for ninja and WebKit) removes the compile from the bake. That needs a maintainer.