test: [de-fork] trace-golden e2e harness (session trace as behavioral oracle)

Closed
AN
1mo ago
+4011-1

PINEAPPLE

Summary

A novel integration-test technique for de-fork stages S5 (native/plugin tool parity) and S7 (validator continuation): drive a real headless session with a scripted model, capture the trace it emits, normalize out nondeterminism, and partial-order-match it against a committed golden. Test infra only — no product-code change.

Explicitly NOT a security oracle: tracing is best-effort/failure-suppressing and TraceSpan has no deny state, so an absent span is not proof of non-execution. (S3's HardPolicy uses an independent audit probe instead.) This harness corroborates observable behavior only.

  • normalize.ts — versioned allowlist-projection normalizer; every field classified stable-keep / volatile-drop / volatile-canonicalize; throws on an unknown field so schema drift surfaces. Structural, timing-independent sibling ranking; callId uniqueness canonicalized; loop-summary hashes stripped structurally; realpath-variant path scrubbing (no macOS /private<HOME> leak).
  • match.ts — partial-order structural diff; concurrent siblings compared as an unordered multiset via full canonical signature.
  • driver.ts — drives a real opencode run subprocess against the fork's own TestLLMServer.
  • trace-golden.test.ts — 34 tests; each reviewed correctness fix has a describe block that fails on revert. Missing golden fails unless TRACE_GOLDEN_UPDATE=1 (rejected under CI). Scenario auto-discovery.
  • stability-check.ts — 100/100 independent subprocess runs → 1 unique hash.
  • fixtures/real-traces/ — 8 benign smoke-session traces (PONG/BANANA/6×7; no PII, only model-version strings) pinning the normalizer against real schema shapes.

This went through three review rounds (a Codex technique review + a Codex code review + my verification) that took it from a plausible-but-flawed oracle → concurrency-safe → 7 correctness blockers fixed. All verified.

Test Plan

  • bun test test/altimate/trace-golden/: 34 pass / 0 fail (verified).
  • stability-check.ts: 100/100 runs, 1 unique hash, matches golden.
  • bun run typecheck: clean.

The CI wiring that runs the live subprocess test lands in a combined "wire de-fork CI checks" follow-up (that ci.yml region is co-edited by a sibling stage).

Checklist

  • Tests added/updated
  • Documentation updated (technique doc)
  • CHANGELOG updated — N/A (internal test infra)

Closes #1012

Summary by cubic

Adds a trace-golden E2E harness for de-fork S5 (tool parity) and S7 (validator continuation). Runs a real headless opencode run session with a scripted model, normalizes the trace, and partial-order-matches it to a committed golden to catch behavioral drift. Wired into CI; test infra only, no product code changes. Not a security oracle.

  • New Features

    • normalize.ts: versioned allowlist projection; drops/canonicalizes volatile fields; rejects unknown keys; stable sibling ranking; path scrubbing.
    • match.ts: partial-order structural diff with unordered multiset compare for concurrent siblings.
    • driver.ts + trace-golden.test.ts: launch real opencode CLI against TestLLMServer; auto-discover scenarios; update goldens via TRACE_GOLDEN_UPDATE=1; 35 tests. stability-check.ts: 100/100 runs produce one normalized hash.

  • Bug Fixes

    • Sanitized fixture home paths to /home/developer/... for portability.
    • stability-check.ts: validate CONCURRENCY like RUNS and fail early on bad input.
    • CI: added test/altimate/trace-golden/ to SUBPROCESS_PATHS so the live golden test runs in the subprocess pass.
    • normalize.ts: narrowed temp-path canonicalization to only mkdtemp-style random tails (preserves distinct stable prefixes; supports / and \) to avoid false matches.
    • Scenario runner: resolve placeholders in prompt and setup files (not just model-script.json) to prevent leaking literal placeholders into the CLI.

Summary by CodeRabbit

  • New Features
    • Added end-to-end “trace-golden” validation for scripted sessions, including deterministic trace normalization and structural golden comparisons.
    • Added a new smoke scenario with representative trace fixtures (success, tool calls, and crash/interrupt cases).
    • Added a stability check that re-runs scenarios in isolated subprocesses to confirm golden consistency.
  • Documentation
    • Expanded trace-golden documentation, including comparison expectations, normalization rules, and guidance for reviewing and updating goldens.
  • Testing
    • Introduced an automated subprocess scenario driver and improved diff output for trace mismatches.