Surface backend preview logs in AI eval failures

Merged
RI
10h ago
+147-2

Summary

  • add a small formatter that promotes backend preview logs/result excerpts into failed AI eval check details
  • use it for script and flow backend-preview failures while preserving the full backend-preview.json artifact
  • add regression coverage for stacktrace/result excerpts and bounded noisy logs

Related to #9296.

Why

Windmill already captures backend preview job logs in artifacts, but the failed benchmark check only showed the workspace/job id. For agent-generation regressions, especially CRUD/app failures, the stacktrace is the signal the model and maintainer need first. This keeps the artifact as the source of truth while making failures immediately actionable in the eval summary.

Verification

  • git diff --check
  • manual Node TS import smoke test for formatPreviewFailureDetails

I could not run the Bun test suite locally because this environment does not have bun, npm, or pnpm installed. The PR includes ai_evals/core/previewFailureDetails.test.ts for CI.

Screenshots

Both screenshots show the same real failed preview job. A generated Python script installs requests, prints two lines, then raises KeyError: 'customer'. The job ran on a local backend through the eval's own BackendPreviewClient (runScriptPreview). The check is built exactly as ai_evals/modes/script.ts builds it on main and on this branch. Each screenshot prints the failed check's details with jq -r from the result JSON. The machine's hostname is redacted as <host>.

Before (main): the failed check only names the workspace and the job.

9383-before

After (this PR): the check carries the tail of the job log and the error with its traceback. Line breaks are kept, so the frames and the ^^^ markers under the failing expression read as they do in the job log.

9383-after


Summary by cubic

Surfaces backend preview logs and result snippets in failed eval check details for script and flow modes, making failures actionable at a glance. Supports Linear #9296 by exposing stacktraces in the eval summary while keeping backend-preview.json intact.

  • New Features
    • Added formatPreviewFailureDetails to format workspace, job ID, and truncated log/result excerpts (default 1200 chars).
    • Log excerpts keep the job log's tail so the error survives install noise; result excerpts lead with the error message.
    • Excerpts preserve line breaks (normalizing CRLF, trailing whitespace, and collapsed blanks) so tracebacks and ^^^ markers read correctly.
    • Integrated into script and flow mode runners; added unit tests for excerpts and truncation.