Introduce conformance test CLI

Draft
HE
1mo ago
+5318-36

Build extensible CLI to host gvisor and LTP (and potentially much more) conformance test suites


Summary by cubic

Introduces a conformance test harness that runs registered Linux suites (gvisor, LTP, and more) against elfuse and a QEMU reference, judging results against JSONC expectations. The scripts/conformance CLI has a fixed capability-first command tree that suite registration cannot extend.

New Features

  • Adds scripts/conformance commands for listing suites, enumerating cases, running lanes, seeding expectations, writing reports, and cleaning up after interrupted runs.
  • Adds elfuse and QEMU backends with process capture, timeouts, and signal classification.
  • Adds JSONC expectation files to declare passes, failures, skips, and quarantines.
  • Adds payload pinning to lock suite artifacts to content hashes.
  • Adds a CI workflow running PR-scoped and full scheduled conformance runs.

Other Changes

  • Updates the QEMU runner to write a state file, fail when guest /tmp setup fails, arm its cleanup per CLI command, and verify a recorded pid still belongs to the VM before stopping; the shared ssh options now live in tests/lib/qemu-ssh.sh.
  • Adds make clean-conformance to sweep leftover VMs, guest processes, and scratch after interrupted runs; lanes record the shared memory segments they create so cleanup removes those and reports the rest.
  • Wires the harness into make check via selftests and adds docs/conformance.md plus the command reference in docs/testing.md.