Apply the shared eval contract across existing harnesses

Merged
MI
2mo ago
+3428-271

Applies the shared eval contract to Claude Code, Codex, Deepagents, Eve, FX, Mastra, and Pi. The autonomy policy uses an additive native system channel where available and one task prefix otherwise, preserving each harness's stock instructions.

Forwards session loss, retained evidence, usage-presence flags, and the actual budget unit into the shared runner. Pi reasoning follows explicit override, model suffix, then SDK default; Codex forwards the requested reasoning effort even with a caller-owned SDK. Pi aliases use the canonical model for token-price computation, and console diagnostics survive normal log persistence. Eve rejects malformed provider/model IDs before execution. FX totals count cached input once: 1,000 input tokens, including 600 cached, plus 200 output and 50 reasoning tokens total 1,250. Native Anthropic thinking blocks are retained as reasoning without appearing in visible Deepagents output.

Validation: 88 focused Codex/Eve/Pi TypeScript tests across six files and 32 Deepagents Python tests pass. Eight FX runner tests also pass, including the cached-input total regression. A separate 12-test Eve run passes with both budget overrides set in the invoking environment. The final consolidated build and workspace typechecks pass with the FX correction, along with all 1,006 eval tests and 769 runtime tests. Existing Pi loader regressions cover suffix/override precedence. Provider responses are local fixtures; this does not establish live model performance or disconnect recovery.

Uses @openai/codex-sdk 0.153.4 with its bundled CLI, which supports max, ultra, and persistent reasoning effort. The allowlist is checked against the SDK type, and invalid effort is rejected before browser startup. No EVAL_CODEX_PATH override is needed for these values.

Validation for the SDK update: 30 Codex runner/adapter/bridge tests and 26 SDK session tests pass, with eval and SDK typechecks. A live smoke run at max completed in 14 seconds using the bundled CLI and local Chrome: the agent used the mounted facade, returned the Example Domain heading and IANA link, and produced browser evidence.

Stack base: evals/consolidation-12-shared-runner-contract. Review this PR relative to its immediate predecessor.

Reviewer entry points:

Current open stack: #2893 → #2894 → #2902 → #2903 → #2889 → #2904 → #2905 → #2906 → #2907.