feat: harness reliability — run termination protocol, context-safety margins, compaction fidelity

Merged
AN
2mo ago
+12956-602

Issue for this PR

Closes #1170

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Evidence-driven reliability improvements to the agent run harness — the loop that decides when a session compacts, when it terminates, how tool output is truncated, and how a run invocation reports what actually happened. These were derived from analyzing a corpus of failed/lost agent sessions and grouped into two waves:

Wave 1 — structural fixes (summarizer integrity, truncation, id sanitation, honest accounting)

  • compaction.ts: the post-compaction continue-message now carries format/tools/system/variant like the normal replay branch, so auto-compaction no longer silently widens the permission surface. The summarizer is called with explicit toolChoice: "none" plus an empty-summary retry-once-then-error guard, which prevents post-compaction amnesia caused by tool-call-shaped summaries.
  • llm.ts: skip stub-tool injection when a request declares zero real tools (summarizer fallback path).
  • truncate.ts / truncation.ts: bash output now middle-truncates (1/3 head + 2/3 tail) via a shared truncate-core.ts, so both leading first-errors and trailing verdict lines survive; the two near-duplicate truncation modules were deduped onto one core.
  • processor.ts / message-v2.ts: deterministic sanitation of malformed (non-string) tool-call ids, with atomic call/result pair aliasing at ingestion and replay.
  • run.ts: turnCount now excludes compaction-machinery steps; error serialization is never an empty {}; the process exits nonzero on fatal abort; provider 5xx/timeout gets a bounded, logged retry; run output carries dual-attribution termination fields (why_model_stopped / why_harness_stopped).
  • Two supporting fixes folded in: a head-truncation fallback that summarizes what fits instead of killing a session outright when a single oversized tool result overflows the context window between turns, and turn-boundary-aware truncation (a head cut that starts mid-turn was getting rejected by providers with a 400, defeating the fallback).

Wave 2 — core-loop fixes (termination path, task pinning, facts ledger, starvation breaker, nudge arbiter)

  • session/termination.ts + processor.ts + cli/cmd/idle-done.ts: explicit DONE-token termination (the harness no longer trusts a bare provider finish-stop as "done"); a run-mode-only idle-done fallback with build-after-last-write ordering and a one-shot confirm-DONE challenge (recursion-guarded); a done_reason field is now emitted on every run.
  • session/prompt.ts + compaction.ts: the original task instruction is now pinned verbatim through every compaction cycle (mode-aware selection between CLI run-mode and interactive sessions, a dynamic size cap with a livelock guard, and a deterministic "contract card" of extracted literals) — this stops the agent losing or hallucinating literal task details (table names, file paths) once the task itself has scrolled out of the summarized history.
  • compaction.ts: a deterministic, append-only corroborated-facts ledger carried across continue-messages, plus first-person summary framing.
  • session/starvation.ts + session/nudge.ts: a write-starvation circuit breaker (annotate-only by default, config-armable), repeat-signature loop detection, a doom-loop guard, and a single-directive nudge arbiter that resolves conflicts between termination, breaker, and budget nudges by explicit precedence instead of whichever fires last.
  • packages/core config schema: all of the above thresholds (starvation breaker mode/limits, idle-done gating, task-pin sizing) are config-exposed knobs with documented default provenance, not hardcoded constants.

Also included: a proactive overflow-estimation fix (the overflow check now accounts for tool output appended since the last recorded token usage, so compaction triggers before a request bounces off the context wall instead of after) and a small addition to the builder agent's prompt — a mandatory finish protocol (re-check the task's literal contract, run a final build so the manifest reflects every change, and stop exploring/commit when turns are running low).

Wave 3 — context estimator safety margin, per-tool-result dispatch cap, run-mode default

  • compaction.ts: the overflow check now triggers against an effective context limit (base * context_safety_fraction, default 0.65, config-exposed as compaction.context_safety_fraction / env ALTIMATE_CONTEXT_SAFETY_FRACTION, with a 4000-token floor) rather than the raw declared limit. The char-based token estimator undercounts real tokenization of dense, structured tool output by a material margin, and compaction previously fired too late to prevent an actual provider-side context-overflow error on that class of content; the safety margin absorbs the worst observed undercount.
  • New tool-result-cap.ts: a hard dispatch-time cap on every individual tool result (min(configured dispatch_max_tokens, byte-derived cap, 15% of effective limit), with middle truncation and long-line chunking), enforced in processor.ts before persistence. This closes a bypass where a single oversized tool result (e.g. one large query result set) could jump a small conversation past the context wall in one step, before the overflow check on the next turn ever ran.
  • run.ts + new run/run-mode.ts: the run CLI command now implies ALTIMATE_RUN_MODE=1 by default (an explicit 0/false is preserved as an opt-out), so any external driver invoking run gets the run-mode termination semantics without needing to set the environment variable itself. Interactive/TUI behavior is unchanged.
  • config.ts: adds the compaction.context_safety_fraction and tool_output.dispatch_max_tokens schema keys.
  • 32 new tests across 3 suites (worst-case-fits proof for the safety margin, giant-tool-result replay, run-mode opt-out behavior).

Interactive TUI behavior is unchanged — all of the run-mode-specific behavior (idle-done fallback, task-pin mode selection, the Wave 3 run-mode default) is gated on the existing run-mode/non-interactive signal and was verified not to fire in interactive sessions.

Pre-PR adversarial review: before opening this PR, the full changeset went through an adversarial review pass looking specifically for correctness edge cases in the new termination/compaction/idle-done logic. That review found 5 high-severity issues, all fixed here: a termination false-positive (the DONE detector could fire on a code-fenced, inline, quoted, or indented occurrence of the token rather than requiring a standalone final line); a livelock at the task-pin/compaction threshold boundary (the pin budget and the overflow check computed their effective limits independently and could disagree at the edge); the idle-done fallback not honoring an explicit opt-out; a challenge-send failure being silently swallowed instead of propagating as fatal; and a fitHead budget calculation that didn't share the same effective-limit path as the rest of compaction. 6 additional medium/low findings from the same review were also fixed directly. 7 remaining deferred medium-severity findings — judged non-blocking for this PR — are tracked in .github/meta/harness-review-followups.md. This pass also included a sweep of code comments to remove internal-process references (planning-document shorthand, corpus statistics) that had leaked into shipped source comments; nothing in the sweep changed behavior.

How did you verify your code works?

  • Typecheck: bun run typecheck clean in both packages/opencode and packages/core.
  • Unit/integration tests: 350+ new/changed tests across test/session/, test/tool/, test/cli/, and packages/core/test/config/ covering the new modules (termination.ts, starvation.ts, nudge.ts, idle-done.ts, run-accounting.ts, truncate-core.ts, tool-result-cap.ts, run/run-mode.ts) and the modified compaction/processor/prompt/run/config paths, run via bun test.
  • Upstream marker check: bun run script/upstream/analyze.ts --markers --base main --strict — clean, no unmarked changes to upstream-shared files.
  • Paired evaluation methodology: Waves 1+2 were validated with a dual-lane paired protocol — the same task set run with and without the harness changes, 3 seeds per task, split across a frozen task set and a held-out task set, to separate genuine reliability improvement from seed noise or task-set overfitting. That protocol measured the clean-exit rate (a session ending via explicit termination rather than crash/timeout/context-death) improving from roughly 15% to roughly 50% on the frozen set. The equivalent fleet dual-lane 3-seed validation has been RESTARTED on the hardened binary (post adversarial-review fixes) and is IN PROGRESS — no pass/fail claim is made for the combined Wave 1–3 + hardening changeset's measured impact on clean-exit rate.
  • Cloud-model smoke test: PASS — a smoke run against a hosted cloud model provider (not the local evaluation harness) completed with clean, explicit termination and no anomalies.
  • What was NOT verified: the held-out-set numbers from the Waves 1+2 paired evaluation, and the restarted fleet dual-lane results for the full changeset, are not included here pending completion. Load/soak behavior under sustained production traffic has not been exercised. TUI interactive-mode regression testing was manual spot-checking, not an automated suite.

Screenshots / recordings

Not applicable — this is a non-UI change to the session/run harness.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

[!NOTE]
High Risk
Changes core run termination, compaction overflow/pin math, and prompt retry idempotency—bugs could duplicate tasks, false-complete runs, or lose session state across compaction.

Overview
Hardens headless run and compaction so long sessions can finish honestly, survive context pressure, and keep literal task requirements across summarization.

run CLI now defaults to run mode (run/run-mode.ts, ALTIMATE_RUN_MODE), validates --max-turns, and wires dual-attribution termination (run-accounting.ts: why_model_stopped, why_harness_stopped, done_reason). The event loop gains bounded provider retries with stable messageID and acceptance probes (retry only when the server definitively did not persist the prompt), explicit SSE abort lifetimes, and a nonzero exit on fatal aborts. Idle-done (idle-done.ts) is a run-mode-only fallback: after green verify strictly after the last mutation, it issues a one-shot confirm-DONE challenge (with continuation if declined), using stricter bash classify/mutation rules.

Compaction (compaction.ts) adds a context safety fraction for estimate-based overflow, head fitHead truncation with telemetry, a redacted state ledger and summary carry built from full session history, verbatim task pinning with livelock halving, summarizer toolChoice: "none" plus empty-summary retry/error, and post-compaction continue messages that preserve user message metadata and carry ledger/nudge text in run mode.

Config (packages/core): V1/V2 schema and ConfigMigrateV1 carry tail_turns, starvation breaker, dispatch caps, and compaction knobs; tests cover mixed V1/V2 documents and round-trips. Builder prompt adds a mandatory finish protocol. .github/meta/harness-review-followups.md lists deferred review items unchanged on this branch.

Reviewed by Cursor Bugbot for commit dda3a2deb4ffee2dd9d17b54e0ce822fb4381b58. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Hardens the headless run harness and session compaction so runs no longer treat a provider stop as completion, lose task state across compaction, or fail when a large tool result overruns the context window. Runs now require an explicit final-line DONE (run-mode only — interactive sessions never see or emit the literal token), recover oversized prompts with bounded summaries and truncation, and report fatal aborts with a nonzero status. Closes #1170. On the frozen Waves 1+2 task set, clean exits improved from roughly 15% to 50%; no result is claimed yet for the full post-hardening changeset.

Run reliability

  • Bare provider finish-stop no longer ends a run; DONE must be a standalone final plaintext line under CommonMark fence rules, and the completion instruction is injected only when run-mode headless with the builder agent so interactive chat is unaffected.
  • run enables run mode by default, while explicit ALTIMATE_RUN_MODE=0 or false opts out and interactive TUI/serve behavior stays unchanged, including nested sessions launched from the bash tool.
  • The idle-done fallback requires a successful verifier after the last mutation, recognizes common mutating command forms (including sed --in-place and space-separated substitutions), and sends at most one confirm-DONE challenge.
  • Turn budgets exclude compaction steps, fatal aborts return nonzero, termination records model and harness reasons, and provider retries use stable message IDs with fail-closed acceptance checks.
  • Starvation, repeat-loop, doom-loop, and nudge protections prevent spinning sessions; repeat detection counts changing tool output as progress, and the starvation breaker remains annotation-only by default.

Compaction and context safety

  • The current run's task stays pinned verbatim through compaction, while an append-only, redacted facts ledger reads the unfiltered session history; observation masks replayed in place of cleared tool output are redacted for secrets before replay.
  • Estimated overflow uses a configurable safety fraction (default 0.65, with a 4000-token floor) that also scales the unknown-model fallback cap and counts tool output added since the last usage reading; exact provider usage keeps the raw limit.
  • Oversized successful or failed tool results are capped and middle-truncated with outcome-aware hints, and the fallback summarizer drops only complete user turns instead of terminating the session.
  • Summaries preserve request metadata, use an explicit no-tool mode with empty-summary protection, and malformed tool-call IDs are sanitized consistently during ingestion and replay.
  • V1 and V2 schemas expose and migrate the compaction, task-pin, starvation, and tool_output.dispatch_max_tokens controls; affected typecheck, marker, and regression suites pass.

Summary by CodeRabbit

  • New Features
    • Added configurable context safety, task pinning, state tracking, summary carry-forward, and per-result output limits.
    • Added run-mode safeguards for stalled, repetitive, or incomplete sessions, with clearer completion and termination handling.
    • Added telemetry for compaction and session reliability events.
    • Tool output truncation now preserves leading errors and trailing results by default.
  • Bug Fixes
    • Malformed tool-call IDs are normalized consistently.
    • Fatal run errors now return a nonzero exit status.
    • Historical tool references no longer create unnecessary placeholders when real tools are available.