[BREAKING][eval][cli] fix: harden source digest, rubric parsing, and secret redaction

Merged
JO
4d ago
+3085-284

What

Known secrets can reappear in API diagnostics and CLI output after JSON escaping, Python repr, whitespace normalization, response conversion, or exception chaining. This change uses one shared redactor at those presentation boundaries, while preserving successful SDK response data and protocol error classification.

provided secrets + ambient credentials
  -> shared known-value redactor
  -> platform diagnostics, CLI output, local eval logs and bridge errors
  • Match literal, trimmed, multiline, JSON-escaped, Unicode-escaped, and Python-repr spellings; sanitize structured diagnostic values and prevent replacement markers from recreating secrets.
  • Pass secret context through secret set, train/eval/benchmark submission, and eval retry. Sanitize HTTP diagnostics before display formatting, suppress sensitive exception chains and opaque response fallbacks, validate successful response types, and retain authentication/billing/conflict classification.
  • Scrub CLI success results, warnings, error details, and debug traces, including output after plain normalization or Rich rendering. Inspect Text base, span, and named theme styles for hidden hyperlink targets across console output, URL printing, table cells, and the command renderer. Remove unsafe links and styles while retaining scrubbed visible text, safe links, and the caller's original objects. Use fixed diagnostics if error reporting itself fails.
  • Cover local eval preflight, child and structured logs, runner errors, replayed diagnostic projections, and upload output. Guard bridge response conversion and serialization in JSON, SSE, and keepalive paths; make provider error text UTF-8 encodable.
  • Length-prefix source digest fields to prevent ambiguous path/content concatenation. Skip non-object rubric messages and report the normal line-numbered error when no usable assistant response remains.

Why

The old digest hashed {"ab": b"c"} and {"a": b"bc"} identically, allowing stale bundle reuse or an incorrect resume match. Literal-only redaction also missed recoverable spellings and outputs outside the original platform error handlers.

Before/after probes against the prior PR head confirmed five redaction gaps: numeric JSON values, trimmed credentials, JSON escaping, individual multiline components, and Python repr. Each remained visible before and is redacted now. Regression tests also cover ordering and overlapping secrets, protocol preservation, malformed responses, rendered output, and failures inside error handling.

Compatibility and risk: Source digests change for every project. Bundle caches rebuild once; named local eval runs created with the previous digest, including SDK 0.3.8, refuse resume and --retry-failed. Finish them with their original SDK, choose a new run name, or use --fresh to archive the old results. Existing manifest and journal bytes are not migrated automatically; completed results remain available for inspection and upload. See upgrade guidance.

The patch is reversible, but run fingerprints are version-dependent. Known-value redaction can hide extra diagnostic context, especially for short secret components. It does not decode arbitrary third-party transformations or sanitize user-authored datasets, trajectories, artifacts, or old journals; SDK-authored prebuilt Rich tables and panels are not recursively rescanned by the console; Text values and cells passed through the console facade and command renderer are checked.

How to Test

  • Local validation on e16f626e: 2,289 tests passed on Python 3.13, serial execution, covering CLI, platform, auth, shared redaction, public imports, and documentation. The 16 initial Rich hyperlink regression cases failed before the fix; all 22 final cases, including named theme styles, pass. They inspect actual OSC 8 terminal output, verify safe links remain clickable, and check that input Text objects are unchanged.
  • ruff check --no-cache ., ruff format --check --no-cache ., ty check --error-on-warning osmosis_ai/, and git diff --check passed. All applicable pre-commit hooks, including private-key detection, passed.
  • CI for e16f626e passed: lint, ty/public API Pyright verification (with the existing Harbor baseline), complete Python 3.12/3.13/3.14 test suites including coverage, both Harbor compatibility variants, package build, and all nine isolated wheel smoke scenarios.
  • No local production build or live provider/sandbox acceptance run was performed.
Exact local test command 
PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True .venv/bin/python -m pytest -q -p no:cacheprovider --no-cov --tb=short tests/unit/cli tests/unit/platform tests/unit/auth tests/unit/test_redaction.py tests/unit/test_public_api_imports.py tests/unit/docs

Checklist

  • PR title follows [module] type: description format, with [BREAKING] for the source-digest resume impact.
  • ruff check . and ruff format --check . pass.
  • ty check --error-on-warning osmosis_ai/ passes; public API Pyright verification has no unexpected errors.
  • Full pytest matrix passes on Python 3.12, 3.13, and 3.14.
  • Public behavior and compatibility changes are documented.
  • No real secrets or credentials included; regression fixtures use synthetic values.