fix(tracing): close trace corruption in long-running sessions (#865)
Summary
Closes #865.
Two confirmed concurrency bugs in the trace pipeline that caused trace files in long-running sessions to either lose events (M2) or show a stale terminating status instead of "crashed" (M3). Both fire 100% of the time under realistic conditions in the reproducers added in this PR. The third architecturally-related bug (M1) remains as a documented design hazard; commit 38463876b's existing protection holds against it on local SSD.
See #865 for the full reproduction matrix and mechanism writeups.
What changed
M2 — debounce drops events with no follow-up snapshot
Trace.snapshot() returned early when snapshotPending=true and was never re-scheduled after the in-flight write completed. In bursty turns (an LLM step firing several tool calls back-to-back), the disk file lagged memory until a fresh event arrived in a future turn. If the process exited in the gap, the burst's tail was lost. Natural reproducer fired 20/20 trials, losing 7 of 8 burst events each.
Fix: track snapshotRequestedDuringPending whenever snapshot() short-circuits. In .finally(), if the flag is set and the trace is neither crashed nor ending, schedule exactly one follow-up snapshot via queueMicrotask. Bounded — at most one extra write per "burst → quiet" cycle regardless of burst size.
M3 — FileExporter.export() races flushSync with no crashed guard
endTrace() calls exporter.export(), which does writeFile(tmp) → rename(tmp, final). On a multi-MB trace the writeFile takes 100+ms, a wide window for flushSync to interleave. The crashed flag added in 38463876b only guarded Trace.snapshot(), not FileExporter.export(), so flushSync's synchronous crashed write was overwritten by the export's rename. Natural reproducer fired 10/10 trials with ~52 MB traces.
Fix: give FileExporter a private _crashed flag with markCrashed(). Check it at entry, before the writeFile, and before the rename (drop tmp + bail). flushSync iterates exporters and calls markCrashed() on each FileExporter before its own synchronous write.
M2 companion — endTraceStarted gate
Once endTrace() claims the canonical write, no concurrent snapshot may run. Without this, the M2 follow-up's queued microtask runs after await snapshotPromise in endTrace but before the root-span endTime mutation, capturing pre-end state and racing to be last writer.
M1 — NOT closed in this PR
The TOCTOU between the synchronous if (this.crashed) check and the asynchronous kernel fs.rename syscall remains. The existing crashed flag from 38463876b protects against it on local SSD (microsecond rename window) and is verified by M1-natural in the new test file (5/5 trials still preserved crashed status under preload + flushSync). Theoretical exposure remains on slow/network filesystems. Documented as a known design hazard in #865; deterministic reproducer kept as test.skip for local experimentation.
Tests
- New:
packages/opencode/test/altimate/tracing-rename-race.test.ts— reproducers + regressions for M1/M2/M3.M2-natural: was 20/20 drops → 0/20 after fixM3-natural: was 10/10 clobbered → 0/10 after fixM2 deterministic(injected fs.rename delay): also passes after fix (extended wait for follow-up snapshot)M1-natural: regression against existingcrashedprotection (0/5 clobbered)M1 residual:test.skip— documents the architectural TOCTOU
- Updated:
packages/opencode/test/altimate/tracing-display-crash.test.ts:500. The pre-existing testflushSync then endTrace — endTrace overwrites crashed statusasserted the opposite of the new policy. Updated to assert thatflushSync's crashed status is preserved (M3 fix's intent).
Verification
- Full opencode test suite: 365 pass, 40 skip, 0 fail in the tracing dir (14 test files).
- Reduces baseline suite failures from 10 → 5; the 5 remaining failures are pre-existing in unrelated areas (
tool.registry,sql_analyze, validator E2E) and unchanged by this PR.
Test plan
- All tracing tests pass locally
- Natural reproducers (M2, M3) confirm 0 hits after fix
- M2 deterministic test passes with extended wait
- M1 protection regression test still passes (existing
crashedflag holds) - No new failures elsewhere in the suite
🤖 Generated with Claude Code
Summary by cubic
Fixes trace corruption in long-running sessions by preserving crash writes and catching up dropped snapshots. Also corrects a failing sql_analyze test assertion that was blocking CI.
- Bug Fixes
- M2:
Trace.snapshot()now schedules one follow-up when a call is dropped during an in-flight write;flush()drains this follow-up so disk matches memory, and snapshots stop onceendTrace()begins. - M3:
FileExporteradds per-sessionmarkCrashed(sessionId)(with sanitization) and checks at entry, pre-write, and pre-rename;TraceExportersupports optionalmarkCrashed; shared-exporter regression fixed so one session’s crash doesn’t suppress others;endTrace()returns the crashed file path when a crash occurs. - Tests: S27
sql_analyzeassertion fixed to not expectschema_inspectwhen an error is returned, unblocking CI.
- M2:
Summary by CodeRabbit
-
Bug Fixes
- Crash exports are preserved: synchronous crash writes are no longer clobbered by concurrent async exports; finalization suppresses further async snapshots and returns the preserved crash output.
- Snapshot concurrency hardened so snapshot requests during in-flight writes are queued and flushed before completion.
-
Tests
- Expanded regression suite and updated tests cover rename/race scenarios, flushSync behavior, cross-session exporter cases, timing variations, and a real-run failure expectation.