fix(plugin): make a Laminar trace red iff the turn failed

Merged
AL
3mo ago
+39-1

Problem

Laminar trace status for v4 runs is uninformative in both directions.

Green when the run failed. Staging run fb5dad0c (session 2cbca552, ENG-5640) failed on its 6th LLM step when the provider stream went silent mid-body and the control plane sent a terminal event: error frame. Laminar trace dc9b5851 reports status: success, 0 errored spans — and the failing session.llm / ai.streamText.doStream spans are absent entirely: the turn span ends 00:22:34.59 while its last child ends 00:21:02.13.

The AI SDK opens both spans with endWhenDone: false and ends them inside a transform flush (ai/dist/index.mjs:6834, :7494). An error arriving as a stream chunk becomes Effect.fail(event.error) in session/llm/ai-sdk.ts:264, the consumer stops pulling, flush never runs, and the spans are never ended or exported. Nothing else in the pipeline sets ERROR on anything, so the trace reads as a clean, shorter run.

Red when the run succeeded. Tool.wrap uses Effect.orDie (tool/tool.ts:145), so a tool that throws rejects execute(), and the AI SDK calls recordErrorOnSpan on the ai.toolCall span (ai/dist/index.mjs:2845). Laminar sets a trace to error if any of its spans is error, so one buggy LLM-generated browser_execute snippet reports the whole run as a failure. That is not an edge case — prod, 2 days:

1058  browser_execute (TOOL)
  59  apply_patch (TOOL)
  50  webfetch (TOOL)  50  grep (TOOL)  45  read (TOOL)

A typical one is the model writing sizeElements where it defined sizeButtons — the agent reads the tool-error, fixes the snippet, and finishes the task successfully.

Change

Establish one invariant: a trace is red iff the turn failed.

  • processor.ts — demote ERROR to UNSET on ai.toolCall spans in onEnd, preserving the message on a bcode.tool.error attribute and leaving the recorded exception event untouched. Mutated in place because setStatus/setAttribute are no-ops once a span has ended.
  • plugin.ts — mark the turn span ERROR on session.error. SessionProcessor.halt (session/processor.ts:679,693) publishes it immediately before status.set(idle), so the span is still open; the existing session.idle case ends it. MessageAbortedError is excluded — stopping a run is not a failure. The recoverable ContextOverflowError path deliberately publishes nothing, so auto-compaction retries stay green.

Verification

Ran the real OTel SDK + InMemorySpanExporter through this processor with an ai.toolCall span carrying an ERROR status and a recorded exception, plus a turn span failed via the session.error path:

ai.toolCall status=UNSET |  | bcode.tool.error="browser_execute snippet threw: TypeError" | events=1
turn         status=ERROR | APIError: stream inactive >90s | bcode.tool.error=undefined       | events=0

bun typecheck clean in packages/bcode-laminar.

Not addressed

The dropped LLM spans themselves. When a provider error arrives mid-stream, the failing call's tokens, cost, latency and msToFirstChunk are lost because the AI SDK never ends the span — that needs an upstream fix or a patch to ai. This PR makes the failure visible on the turn span; it does not recover the measurements.

Summary by cubic

Make Laminar traces go red only when the turn fails, and stay green when tools error as part of normal recovery. Fixes misleading status in v4 runs and aligns with ENG-5640.

  • Bug Fixes
    • Demote ERROR on ai.toolCall spans to UNSET; preserve the message in bcode.tool.error and keep the recorded exception event.
    • Mark the turn span ERROR on session.error while it’s still open; ignore MessageAbortedError, and keep auto-compaction (ContextOverflowError) retries green.