Harden Codex CUA facade isolation, evidence, and tool logging

Merged
MI
2mo ago
+838-157

Stack

Top-of-stack child of #2906 (evals/consolidation-17-gemini-cua).
Review against that immediate parent, not main.

Summary

Port the remaining focused Codex/Stagehand facade hardening from the experimental
Codex worktree without replacing the consolidation stack's newer shared runtime.

  • Add opt-in facade JSONL tool logging: request/session IDs, actual arguments/code,
    paired starts/ends, timing, errors, bounded result previews and browser readiness.
    Never write logs to MCP stdout; redact known credentials and omit image payloads.
  • Escape Unicode line separators in text tool results, addressing the stream
    parsing failure observed in the Hostelworld benchmark traces.
  • Share isolated HOME/CODEX_HOME creation between the SDK example and evals;
    do not inherit operator plugins/config/thread context. Preserve file-based auth.
  • Abort unexpected MCP servers, await observation capture, and match evidence by
    tool-call ID so unrelated calls/missing final captures do not shift screenshots.
  • Add optional bounded SDK failure artifacts and keep binary payloads out of
    telemetry without mutating original events.
  • Document the difference between Codex event logs and facade server logs, privacy
    limitations, authentication, and runnable logging commands.

This remains the existing Codex SDK harness with the Stagehand facade, not a new
native OpenAI computer-use adapter. Preserve the parent's terminal session-loss
semantics, shared facade API, HardBench dataset, verifier and usage contracts.
The older experimental state/nodeRepl/reset surface and reconnect implementation
are not copied wholesale into the newer shared runtime. The original dirty
worktree remains untouched.

Validation

  • 220 focused tests pass across 23 files (core facade, Codex SDK, eval adapters).
  • 2 standalone Codex example tests pass.
  • Built stdio regression verifies the JSONL file is created and contains paired
    request IDs and error results while the MCP client remains responsive.
  • Integration packages and eval ESM/CLI builds; core/Codex/example/evals typechecks.
  • Targeted lint: no errors; warnings remain in existing patterns and diagnostic
    stringification. git diff --check passes.
  • No paid model or HardBenchmark rerun; no benchmark score improvement claimed.

Operational notes

Logging remains opt-in. Logs/error artifacts may contain sensitive model/page
content despite best-effort redaction and must be reviewed before sharing.
Unexpected-server detection aborts observed calls; it is not a security sandbox.
Keychain-only login is not copied into isolated profiles.


Summary by cubic

Hardens Codex facade runs with isolated HOME/CODEX_HOME profiles, opt-in tool logging, and evidence keyed to tool-call IDs so unmatched or unrelated calls no longer misplace screenshots.

New Features

  • Adds opt-in JSONL facade tool logging with request/session IDs, arguments, timing, errors, and bounded result previews; logs go only to file or stderr, never MCP stdout, with known credentials redacted and image payloads omitted.
  • Saves bounded SDK failure artifacts to STAGEHAND_CODEX_DIAGNOSTICS_DIR or EVAL_CODEX_DIAGNOSTICS_DIR; a failed diagnostic write never masks the original SDK failure.
  • Shares isolated HOME/CODEX_HOME creation between the SDK example and evals via a common helper in @browserbasehq/stagehand-integrations-codex-sdk; only file-based auth.json is copied, never plugins, config, stray CODEX_* env, or inherited thread context, and the example now runs in a temporary working directory and warns Codex to use only the mounted browser tools.

Bug Fixes

  • Matches probe evidence by tool-call ID so missing final captures no longer shift screenshots onto the wrong step.
  • Aborts on unexpected MCP servers (detection only, not a sandbox).
  • Escapes Unicode line separators in text tool results, fixing the stream parsing failure seen in Hostelworld traces.

Current open stack: #2893 → #2894 → #2902 → #2903 → #2889 → #2904 → #2905 → #2906 → #2907.