Improve build system and CI/CD pipeline

This pull request restructures the guest artifact build flow used by semu.

The main reason for this change is the ordering problem in the old CI/CD flow. When a branch was merged into master, the device tests could request the rolling release prebuilt artifacts before the new prebuilt release had been built and published. That meant the tests could boot with stale release artifacts even though the current checkout required new guest artifacts, so the merge-time CI/CD path could fail for reasons unrelated to the emulator changes being tested.

The old PR flow also compared guest artifact inputs against the master branch. When a PR changed a guest config, every later push to that PR could trigger another source rebuild, even if the new commit did not touch any guest config or recipe input. This made PR iteration expensive because the decision was tied to branch drift instead of the exact artifact recipe required by the current checkout.

The old flow also made the build and test pipeline depend heavily on GitHub-specific behavior. Cache restore, workflow artifacts, release downloads, publish decisions, and test setup were coupled together in the workflow. This PR redraws that architecture around clearer layers: source-build recipes, artifact resolution, artifact materialization, packaging/publishing, and test consumption. GitHub Actions remains the current execution environment, but the GitHub-specific pieces are now adapters around those layers rather than the place where the artifact policy lives.

Local development stays close to Make’s file-target model. Missing guest artifacts can still be downloaded from the rolling prebuilt release, while source rebuilds are explicit through make build-artifacts. This keeps the normal local path predictable and preserves the Buildroot cache workflow developers rely on when iterating on guest configuration.

CI owns the artifact decision model. The workflow computes recipe keys for the current checkout, compares them with restored workflow-local raw artifacts and the rolling release manifest, then materializes a plan as cache reuse, release download, or source build. The resolver decides where artifacts should come from, and the materializer performs that plan.

The rolling release contract is also tightened. Recipe keys continue to decide whether a release artifact matches the current source recipe, while archive checksums verify that downloaded .bz2 bytes match the published manifest before decompression and stamping. These checks cover different failure modes, so the release manifest now records both archive checksum entries and virtual recipe-key entries.

Local build flow

make build-artifacts replaces the old image-oriented source build entrypoint with an artifact-class based flow. The local CLI selects which classes to build, handles cleanup policy, and delegates the actual source recipes to shared helpers under scripts/prebuilt/.

The default rootfs and test-tools rootfs now use separate Buildroot output directories. This keeps optional test payloads such as X11 and DirectFB2 from changing the default rootfs.cpio and derived ext4.img path. ext4.img remains a local derived artifact built from rootfs.cpio.

Test-tools payload selection is explicit. Options such as --x11 and --directfb2-test are passed directly to the build command, and DirectFB2 no longer implicitly selects X11.

CI artifact flow

CI is split into resolution and materialization. The resolver reads the current recipe inputs, restored cache stamps, and release manifest, then emits a plan for each artifact class. The materializer executes that plan without making new policy decisions.

Workflow-local raw artifacts are tracked with .stamps/prebuilt-local/*.recipe-key. Those stamps allow restored CI cache entries to be checked against the current checkout before reuse. Test jobs consume either the raw artifact bundle produced by the current workflow run or the rolling prebuilt release through the same composite action.

Publishing is part of the main workflow. After the required tests run against the same materialized artifact set, the publish job verifies the packaged artifacts and updates the rolling prebuilt release when the resolver reports release drift.

Release manifest

prebuilt.sha1 now records both compressed archive checksums and recipe-key entries. The archive checksum entries verify transport integrity for Image.bz2, rootfs.cpio.bz2, and test-tools.img.bz2. The recipe-key entries describe the source recipe that produced each artifact class.

Local release downloads verify archive bytes before decompression. CI release materialization additionally verifies that the manifest recipe key still matches the resolver plan before stamping the raw artifact as current.

Fork and mirror repositories can bootstrap from source when the rolling release manifest returns HTTP 404. Other release download failures remain visible as release or network state problems.

Documentation

The README now describes missing guest artifacts as release downloads and source guest rebuilds as an explicit make build-artifacts operation.

docs/build-system-and-ci.md documents the local build path, CI artifact contract, resolver/materializer model, package and publish flow, and fork/mirror bootstrap behavior.

The networking document is updated to point at the moved device smoke script path.

---

Summary by cubic

Rebuilt the guest artifact build and CI/CD pipeline around artifact classes with a recipe‑keyed resolver/materializer and verified release archive checksums to prevent stale prebuilts and speed up PRs. Added fast build‑system contract tests, selectable tool hooks for reliable ext4.img on macOS, and early validation of default minimal.dtb boot options.

• Refactors

  • Local builds via make build-artifacts backed by scripts/prebuilt/*; separate Buildroot outputs for default rootfs vs. test‑tools; explicit test‑tools flags (--x11, --directfb2-test).
  • CI plans per class (use-action-cache/download-release/build) with stamps in .stamps/prebuilt-local; materializer executes the plan; tests get artifacts via provide-guest-artifacts; removed the standalone prebuilt workflow.
  • Release: prebuilt.sha1 records *.bz2 checksums plus *.recipe-key; downloads verify archive bytes and expected recipe; package/verify via .ci/prebuilt/package.sh and verify-package.sh.
  • Tooling: selectable ROOTFS_CPIO, ROOTFS_CP, ROOTFS_CHOWN, and ROOTFS_FAKEROOT_SHELL for ext4 builds; added scripts/check-rootfs-fakeroot.sh; macOS CI installs Homebrew cpio and sets these envs.

• Migration

  • Developers: make check downloads verified prebuilts; rebuild via make build-artifacts [image|rootfs|test-tools] with --x11/--directfb2-test, --clean-build, or --full-rebuild. Set ROOTFS_CPIO (and related vars) on macOS if needed.
  • Maintainers: rolling prebuilt release publishes from the main workflow on master after tests; master runs aren’t canceled by concurrency.
  • Forks/mirrors: PREBUILT_URL defaults to this repository; CI can bootstrap from source when the release manifest returns 404.