Add OCI image support: pull, unpack, run, prune, status, policy

Closed
MA
4mo ago
+37619-24

This PR lands the full elfuse OCI image support. It supersedes the
original Phase 1 scope of this PR (CLI scaffold + pull/inspect) and
now covers Phases 1-4 plus the post-Phase-3 improvements plan: image
layout alignment, GC/prune, layer + stack snapshot caches, store
status, parallel pull, registry policy.json, and a heavy-mode compat
matrix.

Scope

  • Pull / inspect — content-addressable blob store, HTTPS + bearer
    token, OCI index walk to the linux/arm64 leaf manifest, partial-
    store-aware inspect renderer.
  • Unpack — tar reader (ustar + PAX x/g records), gzip + decode-
    only vendored zstd, whiteout-aware layer apply (typeflag '1'/'2'/'5'
    • .wh.* markers), per-image sysroot on a case-sensitive APFS
      sparsebundle.
  • Run — elfuse oci run clones the unpacked tree via clonefile(2),
    honors Entrypoint / Cmd / Env / WorkingDir / User, and reuses the
    existing elfuse launch path so a dynamically-linked guest binary
    runs through the same shim + syscall surface as the non-OCI mode.
  • Lifecycle — oci prune with --older-than / --keep-bytes;
    layer + stack prune sweep; oci status (text + --json);
    oci rebuild-cache for pre-snapshot stores.
  • Performance — parallel blob fetch with HTTP Range resume;
    per-layer raw snapshot cache; ChainID stack snapshot cache; APFS
    COW clone-rootfs reuse between runs.
  • Policy — podman / skopeo-style policy.json + registries.d
    overlay (per-registry insecure / ca_bundle / auth_file). CLI flags
    override; loopback-only --insecure.
  • Test coverage — 25 OCI unit suites (test-oci-*), compat-shell
    smoke (tests/test-oci-compat.sh), and an opt-in heavy mode
    (OCI_COMPAT_TEST=1) that drives three layered fixtures
    (alpine-shaped, busybox-shaped hardlink dispatch, two-layer
    whiteout) end-to-end through a freshly-provisioned scratch
    sparsebundle.

Summary by cubic

Adds full OCI image lifecycle to elfuse with a new elfuse oci subcommand and a BusyBox‑style elfuse-container alias. Hardened unpack/path containment, verified blobs and sizes, shared VM launch path, and better /dev//proc emulation.

  • New Features

    • CLI: oci pull|inspect|unpack|clone|run|prune|rebuild-cache|status; status --json. Supports elfuse-container (e.g., elfuse-container run alpine:3).
    • Pull: delegates to skopeo copy (auth/TLS/mirrors via containers-policy.json/registries.conf), pins manifests in index.json; multi‑arch defaults to linux/arm64 with optional --arch amd64.
    • Unpack/caches: libarchive tar (ustar/GNU/PAX; gzip/zstd), whiteouts, absolute‑symlink rewrite, parent walks with openat(O_NOFOLLOW), manifest/config re‑hash and layer size checks; APFS case‑sensitive sysroot; raw layer + ChainID stack caches; rebuild-cache.
    • Runtime/launch: shared VM launcher (core/launch.*); PATH resolver; shebang expansion; image User lookup (name/group or numeric); inject /etc/{resolv.conf,hosts,hostname}; improved /dev emulation (/dev/full ENOSPC on write, /dev/console), and basic /proc.

  • Migration

    • Pins live in OCI index.json (store auto‑migrates from refs/).
    • Layer cache schema v2; first open drops legacy v1 entries (blobs/images untouched); symlink rewrite bumps unpack sidecar version and transparently rebuilds caches.
    • Dependencies: use system libarchive and cJSON via pkg-config and require skopeo at runtime (brew install libarchive cjson skopeo or apt-get install libarchive-dev libcjson-dev skopeo).