A failure report that names the failing test

Fixes: Failure report shows only the last 1,024 bytes, which never contains
the failing test.

Branch: relayflow/flows-software-garden-c2deb4cd · Commit: 377b277

The problem, and where the bytes were lost

A deterministic step's failure report cut each captured stream to its last
1,024 bytes. For any runner that streams results and prints its totals last —
TAP (node --test), cargo, vitest, pytest — that last kilobyte is by
construction the summary and the final few passing cases, so the one not ok
was exactly the part that got cut.

The bytes were never lost by the kernel. kernel/relayflowd/src/exec_det.rs
captures the last 64 KiB of each stream, and preserve_failure_output
(kernel/relayflowd-core/src/machine.rs) keeps output intact on a failed
deterministic completion. The SDK read the whole field and then threw it away
at packages/sdk/src/cli/step-failure.ts's tail(). This is a render fix.
No kernel change is made, and none is needed.

What changed

New: packages/sdk/src/cli/step-excerpt.ts (formatStepExcerpt). It
selects three disjoint ranges of the journaled field, in source order, inside a
4,096-byte bound:

• a head of whole lines,
• the lines in the elided middle that match a built-in failure marker,
• a tail that fills whatever the head and highlights did not need.

The elision is stated in band. Markers are matched against an escape-stripped
comparison view of each line, so a coloured FAIL header still matches;
journal data is never mutated, and display sanitising stays a separate, later
step. The marker set is TAP not ok, FAIL, FAILED, a vitest/jest failure
glyph, cargo's ... FAILED and ---- <case> stdout ---- headings, and Rust's
panicked at — each with a test pinning a real line from that runner. They are
hints, not a parser: an unrecognised format still gets head-and-tail context.

Per the plan review, the budget is a fixed 4,096 bytes and the markers are
built in. No environment variable and no failureHighlight spec field are
added here: a parser for a presentation setting that throws while extracting
failure evidence would remove otherwise-readable evidence from the report
(classifyOutcome catches extraction failures and reports Could not inspect the failed step), and a new step key would mean a kernel spec change to solve
a rendering problem.

Header. Stdout (last 1,024 bytes): becomes Stdout (captured excerpt):.
(last 1,024 bytes) was a promise about the excerpt's shape made where it
could no longer be true; the elision marker now makes it in band, only when it
is. Field names stdoutTail/stderrTail are unchanged for compatibility, with
the new semantics documented on them and in docs/SURFACE.md.

What it now prints

A 14-case TAP run with chatty subtest diagnostics, 52,744 bytes captured:

TAP version 13
ok 1 - pty-exit: case 1 passes
# subtest diagnostic 1.0 — the kind a chatty runner prints
        [...]
… 48,816 bytes elided; 1 line matched a failure marker …
not ok 5 - pty-exit: child reaped twice
… end of elided region …
        [...]
1..14
# tests 14
# pass 13
# fail 1

The same stream under the old render — every visible line is ok or a total,
which is the reported bug exactly:

# subtest diagnostic 14.59 — the kind a chatty runner prints
1..14
# tests 14
# pass 13
# fail 1

What this does not promise

Stated here because the excerpt now makes a claim about itself, and the claim
has to be exact.

• It selects from the field the journal carried, which for a deterministic
  step is the last 64 KiB the kernel captured. A failure printed before that
  window is not here to be found.
• The absence of an elision marker means the supplied field fit whole. It
  does not mean the command printed nothing more.
• An agent/llm step's evidence still survives only as the daemon's render
  in verification.detail, capped at 2,000 chars by the daemon. This helps a
  deterministic step fully and an agent step only up to that cap. Raising it is
  a separate change and is out of scope.

Files

File	Change
packages/sdk/src/cli/step-excerpt.ts	new, 317 lines — the whole excerpt contract
packages/sdk/src/cli/step-failure.ts	tail() → formatStepExcerpt; header text
packages/sdk/src/failure-kinds.ts	field comments describe the excerpt
docs/SURFACE.md	"Reading a failed step": grammar, marker set, both limits
packages/sdk/tests/step-failure-excerpt.test.ts	new, 38 unit tests
packages/sdk/tests/step-failure-diagnostic.test.ts	inverted tail assertions; middle-failure through the CLI, human and --json
packages/sdk/tests/authored-run-failure-evidence.test.ts	live-kernel regression
packages/sdk/tests/authored-node-result.test.ts	excerpt across the authored IPC bridge
packages/sdk/tests/{direct-run,resume}-failure.test.ts	header assertions

---

Verification

Every command below was run from packages/sdk, and the output is pasted, not
summarised.

Typecheck and build

npm run typecheck && npm run build && npm run typecheck:tests

> @relayflows/sdk@2.0.25 typecheck
> tsc --noEmit && tsc -p tsconfig.type-tests.json


> @relayflows/sdk@2.0.25 build
> tsc && node scripts/make-cli-executable.mjs


> @relayflows/sdk@2.0.25 typecheck:tests
> tsc -p tsconfig.tests.json

exit=0

The touched tests

npx vitest run tests/step-failure-excerpt.test.ts tests/step-failure-diagnostic.test.ts \
  tests/resume-failure.test.ts tests/direct-run-failure.test.ts \
  tests/authored-run-failure-evidence.test.ts tests/authored-node-result.test.ts

 ✓ tests/authored-node-result.test.ts (39 tests) 15ms
 ✓ tests/step-failure-excerpt.test.ts (38 tests) 46ms
 ✓ tests/step-failure-diagnostic.test.ts (25 tests) 58ms
 ✓ tests/resume-failure.test.ts (2 tests) 7ms
 ✓ tests/direct-run-failure.test.ts (8 tests) 17ms
 ✓ tests/authored-run-failure-evidence.test.ts (9 tests) 2133ms
   ✓ a failing f.run through the live kernel > keeps the evidence when the failure is relabelled as a timeout 508ms
   ✓ the child index after the process that wrote it is gone > still names every child, with its own run id, after a daemon restart 1297ms

 Test Files  6 passed (6)
      Tests  121 passed (121)
   Start at  15:44:39
   Duration  3.24s (transform 1.00s, setup 0ms, collect 3.85s, tests 2.28s, environment 1ms, prepare 271ms)

Full suite — and the failures that are not mine

npm test

 Test Files  7 failed | 154 passed | 3 skipped (164)
      Tests  40 failed | 2556 passed | 25 skipped (2621)
     Errors  1 error
   Start at  15:44:58
   Duration  216.76s (transform 2.43s, setup 0ms, collect 36.89s, tests 560.48s, environment 21ms, prepare 6.51s)

40 tests fail, and all 40 fail identically without this change. They are
environmental in this sandbox: live-kernel, webhook-live,
provider-trigger-executor and mcp spawn a hardcoded
kernel/target/debug/relayflowd, which does not exist because ops/cargo.sh
keeps CARGO_TARGET_DIR outside the repo; stuck-run-triage and
authored-node-runtime fail on expected an @relayflows/surface flow handle;
live-kernel also wants a real analyzer CLI. None of the 40 mentions a step
failure, an excerpt or a stream tail.

Baseline, established by stashing the change, rebuilding and running the same
seven files:

git stash push -u -m excerpt-work -- docs/SURFACE.md packages/sdk/src packages/sdk/tests
npm run build
npx vitest run tests/authored-node-runtime.test.ts tests/communication-mixed-resume.test.ts \
  tests/live-kernel.test.ts tests/mcp.test.ts tests/provider-trigger-executor.test.ts \
  tests/stuck-run-triage.test.ts tests/webhook-live.test.ts

 Test Files  7 failed (7)
      Tests  40 failed | 50 passed | 18 skipped (108)
     Errors  1 error
   Start at  15:42:03
   Duration  64.01s (transform 825ms, setup 0ms, collect 3.04s, tests 123.56s, environment 1ms, prepare 314ms)

The same seven files with the change restored (git stash pop, rebuild), same
command:

 Test Files  7 failed (7)
      Tests  40 failed | 50 passed | 18 skipped (108)
     Errors  1 error
   Start at  15:50:09
   Duration  63.94s (transform 799ms, setup 0ms, collect 3.06s, tests 121.94s, environment 1ms, prepare 282ms)

Identical: 40 failed | 50 passed | 18 skipped (108) either way.

Mutation verification

One claim, done to the letter of AGENTS.md. The change reverted is the
extraction path itself: formatStepExcerpt at the four call sites in
evidence() replaced by the old 1,024-byte tail(). The test is the
live-kernel regression, which runs a real relayflowd, a real command and the
real journal.

Before — the file's hash, then the mutation:

cp src/cli/step-failure.ts /tmp/step-failure.ts.orig && sha256sum src/cli/step-failure.ts

5eb1f42ca884836def6dab61bf5315dabfa22f11e7b9a451adc1f957348538ff  src/cli/step-failure.ts

Mutated — the test fails, at the assertion it exists for:

npx vitest run tests/authored-run-failure-evidence.test.ts \
  -t 'names the failing case from the middle of a real test run'

   × a failing f.run through the live kernel > names the failing case from the middle of a real test run 94ms
     → expected ' realistically long name\nok 206 - pt…' to contain 'not ok 111 - pty-exit: child process …'
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
AssertionError: expected ' realistically long name\nok 206 - pt…' to contain 'not ok 111 - pty-exit: child process …'
 Test Files  1 failed (1)
      Tests  1 failed | 8 skipped (9)

The failing assertion is at tests/authored-run-failure-evidence.test.ts:120,
which is the rendered excerpt. The assertion above it —
expect(journaled).toBe(expected), read back from the child run's journal —
passed under the mutation. That is the bug in one pair of assertions: the
journal held the whole stream and the renderer cut the line out of it.

Restored byte-for-byte, and green again:

cp /tmp/step-failure.ts.orig src/cli/step-failure.ts && sha256sum src/cli/step-failure.ts \
  && cmp /tmp/step-failure.ts.orig src/cli/step-failure.ts && echo "restored byte-for-byte" \
  && npm run build && npx vitest run tests/authored-run-failure-evidence.test.ts \
     -t 'names the failing case from the middle of a real test run'

5eb1f42ca884836def6dab61bf5315dabfa22f11e7b9a451adc1f957348538ff  src/cli/step-failure.ts
restored byte-for-byte

 ✓ tests/authored-run-failure-evidence.test.ts (9 tests | 8 skipped) 89ms

 Test Files  1 passed (1)
      Tests  1 passed | 8 skipped (9)
   Start at  15:44:00
   Duration  89ms

Same hash before and after. Nothing else in this change is labelled
mutation-verified.

One qualification on that hash: two later, cosmetic edits to
step-excerpt.ts (a doc sentence, and sanitising the fits-whole path from
the original string rather than a Buffer round trip) landed after this
mutation run. step-failure.ts — the mutated file — is byte-identical to
the committed version, and the focused suite above was re-run after those
edits. The mutation exercised the extraction path, which those edits did not
touch.

Regression coverage

tests/step-failure-excerpt.test.ts (38 tests) pins the contract: streams
under, at and one byte over the budget; head/highlight/tail selection; byte
accounting (shown + elided == input, asserted arithmetically); one marker
test per supported runner, plain, indented and ANSI-coloured; unknown formats
getting context without claiming a match; head/tail matches not repeated; two
identical lines at two positions counted as two observations; the 20-line cap
with its +N more matching lines count; a matched line too long to fit shown
in part under … line truncated …; CRLF, tabs, control and bidi characters; no
split multibyte characters and no invented U+FFFD while keeping one the capture
itself produced; byte bounds at 256, 1,024, 4,096 and 8,192; and a refusal for
an unusable budget.

The live-kernel regression asserts its own fixture before it asserts the
render: the failure sits past the first 4 KiB and before the last 4 KiB of a
~13 KiB stream, inside the kernel's 64 KiB capture, and is absent from the last
1,024 bytes — so it cannot be satisfied by a larger budget or by head-and-tail
context alone.

Not changed, deliberately

• kernel/ — no behaviour change, so no new Rust tests. If scope ever expands
  to capture, that needs kernel-level regressions of its own.
• cli/status.ts's DETAIL_LIMIT — a different surface with a different
  failure mode.
• Journal selection, retry handling, completionReason, exit status and
  evidence precedence in step-failure.ts are untouched.
• plan.md and reviewed-plan.md are left intact and uncommitted, as is this
  file.

---

[!NOTE]
Low Risk
Render-only SDK change to failure diagnostics; no kernel or persistence behavior change, with broad unit and integration tests.

Overview
Replaces the last 1,024-byte tail of step failure stdout/stderr with formatStepExcerpt, a 4,096-byte captured excerpt that keeps a head, failure-marker lines from the elided middle (TAP not ok, pytest/vitest/cargo/Rust patterns), and a tail, with in-band elision when the journal field does not fit whole.

step-failure.ts routes journal stdout_tail/stderr_tail, worker failure excerpts, and fallback detail through the new formatter; human labels change from (last 1,024 bytes) to (captured excerpt). stdoutTail/stderrTail JSON keys are unchanged; docs/SURFACE.md and failure-kinds.ts document the new semantics and limits (64 KiB kernel capture, no elision marker ⇒ field fit whole).

Tests add step-failure-excerpt.test.ts and extend CLI, live-kernel, and authored IPC coverage so middle-of-stream failures (e.g. TAP) appear in diagnostics instead of only summary lines.

^{Reviewed by Cursor Bugbot for commit a754dc64333d7203b86151f87a9bbbfb02506f6a. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Summary by cubic

Fixes step failure reports so the failing test is no longer cut out of the capture.

• The old render trimmed each stream to its last 1,024 bytes, which for streaming runners (TAP, cargo, vitest, pytest) contained only the summary and passing cases — the one not ok was always the part dropped.
• formatStepExcerpt now prints a head, failure-marker lines from the middle, and a tail within a 4,096-byte budget, stating the elision in band; a stream that fits is printed whole with no marker.
• Failure markers cover TAP not ok, FAIL/FAILED, vitest/jest glyphs, cargo's ... FAILED and ---- <case> stdout ----, and Rust's panicked at; unrecognized formats still get head-and-tail context.
• When the tail's byte cut lands inside a failing line, the excerpt now shows the line from its own start (whole or truncated under … line truncated …) so the test name still appears; a line is counted as shown only when its start is on screen, which also keeps the omission count honest.
• A kept prefix ending exactly where the tail begins is rendered as one continuous run, with no invented cut marker between its two halves.
• Header text changes from (last 1,024 bytes) to (captured excerpt); field names stdoutTail/stderrTail stay unchanged. This is a render-only fix — no kernel change.