fix(tracing): close trace corruption in long-running sessions (#865)

Merged
SA
6mo ago
+667-14

Summary

Closes #865.

Two confirmed concurrency bugs in the trace pipeline that caused trace files in long-running sessions to either lose events (M2) or show a stale terminating status instead of "crashed" (M3). Both fire 100% of the time under realistic conditions in the reproducers added in this PR. The third architecturally-related bug (M1) remains as a documented design hazard; commit 38463876b's existing protection holds against it on local SSD.

See #865 for the full reproduction matrix and mechanism writeups.

What changed

M2 — debounce drops events with no follow-up snapshot

Trace.snapshot() returned early when snapshotPending=true and was never re-scheduled after the in-flight write completed. In bursty turns (an LLM step firing several tool calls back-to-back), the disk file lagged memory until a fresh event arrived in a future turn. If the process exited in the gap, the burst's tail was lost. Natural reproducer fired 20/20 trials, losing 7 of 8 burst events each.

Fix: track snapshotRequestedDuringPending whenever snapshot() short-circuits. In .finally(), if the flag is set and the trace is neither crashed nor ending, schedule exactly one follow-up snapshot via queueMicrotask. Bounded — at most one extra write per "burst → quiet" cycle regardless of burst size.

M3 — FileExporter.export() races flushSync with no crashed guard

endTrace() calls exporter.export(), which does writeFile(tmp) → rename(tmp, final). On a multi-MB trace the writeFile takes 100+ms, a wide window for flushSync to interleave. The crashed flag added in 38463876b only guarded Trace.snapshot(), not FileExporter.export(), so flushSync's synchronous crashed write was overwritten by the export's rename. Natural reproducer fired 10/10 trials with ~52 MB traces.

Fix: give FileExporter a private _crashed flag with markCrashed(). Check it at entry, before the writeFile, and before the rename (drop tmp + bail). flushSync iterates exporters and calls markCrashed() on each FileExporter before its own synchronous write.

M2 companion — endTraceStarted gate

Once endTrace() claims the canonical write, no concurrent snapshot may run. Without this, the M2 follow-up's queued microtask runs after await snapshotPromise in endTrace but before the root-span endTime mutation, capturing pre-end state and racing to be last writer.

M1 — NOT closed in this PR

The TOCTOU between the synchronous if (this.crashed) check and the asynchronous kernel fs.rename syscall remains. The existing crashed flag from 38463876b protects against it on local SSD (microsecond rename window) and is verified by M1-natural in the new test file (5/5 trials still preserved crashed status under preload + flushSync). Theoretical exposure remains on slow/network filesystems. Documented as a known design hazard in #865; deterministic reproducer kept as test.skip for local experimentation.

Tests

  • New: packages/opencode/test/altimate/tracing-rename-race.test.ts — reproducers + regressions for M1/M2/M3.
    • M2-natural: was 20/20 drops → 0/20 after fix
    • M3-natural: was 10/10 clobbered → 0/10 after fix
    • M2 deterministic (injected fs.rename delay): also passes after fix (extended wait for follow-up snapshot)
    • M1-natural: regression against existing crashed protection (0/5 clobbered)
    • M1 residual: test.skip — documents the architectural TOCTOU
  • Updated: packages/opencode/test/altimate/tracing-display-crash.test.ts:500. The pre-existing test flushSync then endTrace — endTrace overwrites crashed status asserted the opposite of the new policy. Updated to assert that flushSync's crashed status is preserved (M3 fix's intent).

Verification

  • Full opencode test suite: 365 pass, 40 skip, 0 fail in the tracing dir (14 test files).
  • Reduces baseline suite failures from 10 → 5; the 5 remaining failures are pre-existing in unrelated areas (tool.registry, sql_analyze, validator E2E) and unchanged by this PR.

Test plan

  • All tracing tests pass locally
  • Natural reproducers (M2, M3) confirm 0 hits after fix
  • M2 deterministic test passes with extended wait
  • M1 protection regression test still passes (existing crashed flag holds)
  • No new failures elsewhere in the suite

🤖 Generated with Claude Code


Summary by cubic

Fixes trace corruption in long-running sessions by preserving crash writes and catching up dropped snapshots. Also corrects a failing sql_analyze test assertion that was blocking CI.

  • Bug Fixes
    • M2: Trace.snapshot() now schedules one follow-up when a call is dropped during an in-flight write; flush() drains this follow-up so disk matches memory, and snapshots stop once endTrace() begins.
    • M3: FileExporter adds per-session markCrashed(sessionId) (with sanitization) and checks at entry, pre-write, and pre-rename; TraceExporter supports optional markCrashed; shared-exporter regression fixed so one session’s crash doesn’t suppress others; endTrace() returns the crashed file path when a crash occurs.
    • Tests: S27 sql_analyze assertion fixed to not expect schema_inspect when an error is returned, unblocking CI.

Summary by CodeRabbit

  • Bug Fixes

    • Crash exports are preserved: synchronous crash writes are no longer clobbered by concurrent async exports; finalization suppresses further async snapshots and returns the preserved crash output.
    • Snapshot concurrency hardened so snapshot requests during in-flight writes are queued and flushed before completion.
  • Tests

    • Expanded regression suite and updated tests cover rename/race scenarios, flushSync behavior, cross-session exporter cases, timing variations, and a real-run failure expectation.