build: Reduce repeated work in the Docker image builds

Merged
SH
2mo ago
+995-456

Stacked: #37344 (syft SBOM) targets this branch. Merge this first.

Summary

The Docker images were built by four near-identical build-push-action steps in
sequence, each using about a quarter of the runner. This moves targets, tags,
pins and build args into one docker/docker-bake.hcl and builds them
concurrently — so CI and pnpm build:docker finally drive the same definition.
The release images were previously produced by a code path nobody ran locally.

Measured on amd64, same runner class:

beforeafter
Release image builds615s140s
amd64 release job17m09s4m48s
Image size2.19GB2.16GB

Largest contributor first: bake parallelism (−406s), dropping the SBOM scan,
relocating the release smoke checks, native modules compiling above
COPY ./compiled, and one copy of the 1.3GB application tree instead of two.
Blacksmith metrics confirm the mechanism — CPU during the build step goes from
~26% to 71%.

The second commit adds the verification the first leans on: a probe asserting the
published manifest is an OCI image index, with a negative control that still
reproduces the 2.26.0 format, and the kafka binding check moved into
smoke-n8n-image.mjs so a broken native build fails on the PR rather than the
next nightly.

Three decisions that look wrong

  • Every job passes the same n8n-io/n8n cache key. Splitting per image was
    the point of the v2 upgrade, but Blacksmith never populates a newly created
    sticky disk — two new keys stayed at 0 bytes across four runs that each wrote
    11GB and logged a successful commit. Splitting would disable layer caching
    outright. Upstream: useblacksmith/setup-docker-builder#125.
  • Two grafts deliberately skip COPY --link. Their destinations traverse
    pnpm symlinks, and a --link copy resolves against an empty root — it replaces
    node_modules/sqlite3 with a directory, with no build error and only a runtime
    failure. Hit and fixed during development.
  • sbom: true is gone, which #31997 added deliberately. It was forcing the
    OCI index format as a side effect; oci-mediatypes=true gets the same format
    without the syft scan, and without the attestation manifests that were
    themselves what containerd misread.

Two smaller trade-offs: the e2e docker-image-size metric stops being emitted
(no images in the daemon, and that image was never the shipped artifact), and the
8vCPU bump is a latency choice at ~2× runner cost — 209s → 175s on the bake step.

How to test

Green on this PR: docker-build-smoke (two-arch, no-cache, all four image smoke
checks), ci-pull-requests (tarball write + shard restore), test-e2e-helm.

docker-build-push.yml has no pull_request trigger, so the release path needs
a dispatch:

gh workflow run docker-build-push.yml --ref devp-docker-bake-consolidation \
  -f push_enabled=true -f include_arm64=false
gh workflow run docker-sbom-probe.yml -f sbom=false

Already run: release build green with pushing; published manifest is
application/vnd.oci.image.index.v1+json with zero attestation manifests; and
the image pulls and serves on AKS — the platform whose containerd failed on
2.26.0 — verified with a staging deploy.

Related Linear tickets, Github issues, and Community forum posts

Follow-up to #31997 (manifest format).

Review / Merge checklist

  • I have seen this code, I have run this code, and I take responsibility for this code.
  • PR title and summary are descriptive. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.
  • PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)