feat(analyst): `answer_commit` renders the final answer deterministically

AN
21h ago
+5171-56

Issue for this PR

Closes #

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why. The analyst's final message was free prose shaped by a prompt paragraph, so the same computed values could be presented differently from run to run: sometimes with the requested fields only, sometimes with extra commentary, excluded candidates or alternative readings. Users and downstream consumers get a consistent final answer only if its rendering is code, not prose.

What changed. A new tool, answer_commit, available only to the analyst agent. The model calls it last with the single chosen value(s) (fields, one field per item the request asks for, in the order asked, so a request for "the repository and the most copied file" carries two fields and a compound request is never collapsed into one), an N-row table (rows) or unresolved with a reason, plus a one-or-two-sentence basis (population, grain, and the source object/column that supplies the requested entity or value) and an answer_only flag. basis is required on every payload, including answer_only and unresolved; it is rendered before the answer line in the normal case and never rendered when the request asks for only the answer. It is the model's reasoning input, not output: live sessions showed that when the basis was skipped, a rule about a related object displaced the requested object's attribution, and that sessions which wrote the basis kept the requested object. Boolean fields accept the string forms "true"/"false" and 1/0, because live Claude sessions on Bedrock send them that way; a validation failure returns a one-line error plus a minimal valid example. The tool validates the payload structurally (exactly one of the three shapes; one value per field; no empty values; rectangular rows; null, "", [] and whitespace fillers from strict-schema clients treated as absent) and returns exactly the rendered final text. Numbers with more than 15 significant digits or outside the safe-integer range are rejected with instructions to pass the exact value as a string, because JSON parsing has already changed them (1234567890123456.7 arrives as …456.8); digits lost before the tool runs (0.10000000000000001 arrives as 0.1) cannot be detected, so the description states the contract: numbers up to 15 significant digits, exact decimals as strings. The complete rendered text is capped at MAX_RENDERED_ANSWER_CHARACTERS (60,000); over the cap the model is told to aggregate, narrow, or write the full table to a file and commit the summary, so an echo can never be truncated at the output limit. basis is bounded by length (400 characters) rather than a punctuation-based sentence count, and the precision range (0 to 10) is stated in the description and the validation message. rows accepts an optional per-column precision map so numeric cells render with fixed decimals (5 → 5.00), after live sessions showed a numeric cell rendered differently from the equivalent string; every precision key must name a column exactly and column labels must be unique when precision is used. rows.format selects the table's text form when the request names one: markdown (default), csv, tsv (header row included, RFC 4180 quoting with embedded quotes doubled, null cells empty, precision honoured) or lines (one row per line, cells separated by one space, no header); with answer_only the reply is exactly the formatted table, so a request for CSV is not answered with a Markdown table that the terminal tool cannot repair. Rendering is a pure function (renderAnswer) with golden tests; markdown table cells are escaped; a single field literally named "Answer"/"Result"/"Value" is not double-labelled. An advisory readings ledger (clause, reading, result summary, decision) is kept in tool metadata and never rendered.

Scoping: the tool is added on the analyst-only gated candidate list, not the global catalogue, so builder, plan, data-explorer and custom agents cannot see it and tool_lookup does not list it; execution also rejects non-analyst callers; a custom agent named analyst receives it by design. The native id is reserved, so an MCP, plugin or file tool with the same name (including case-equivalent spellings on Windows) cannot take the finalizer permission. ALTIMATE_ANSWER_COMMIT=0 removes it. The analyst prompt's final paragraph asks to finish by calling answer_commit with one field per requested item and replying with exactly the returned text, always supplying basis; the previous prose contract (answer-only rule and top-N tie rule included) remains the fallback when the tool is unavailable.

Session loop, scoped to the analyst and marked in session/prompt.ts and session/compaction.ts: (1) once the last answer_commit attempt for the current request has completed, the next generation runs tool-free with the instruction (sent as the final user turn) to reply with exactly the returned text, whether or not a step limit or compaction intervened; a failed attempt, including a failed retry after a success, keeps the tools for repair; a user request that arrives before the echo is sent, or while it streams, clears the commit and is answered normally; the echo generation's text is buffered rather than streamed, and after a stop or length finish the loop publishes one authoritative text part (start, delta, completion) holding the stored committed output, so run, --output and ACP each receive the committed text exactly once and the final text never depends on the model's output budget; one guard finalizes from the stored text whenever the request would otherwise end after a completed commit for any reason other than an abort or a content filter (a failing compaction summariser, a summariser that returns no text twice, a context overflow on the echo, a thrown preparation error, a transient transport failure), recording the cause as assistant-message metadata (answer_commit_echo_error); transport recovery is structural, not phrase-based: only 5xx, 429, 408 or statusless network failures recover, while any 4xx, explicit policy codes (policy_violation, content_filter, content_policy*, guardrail_intervened including nested orchestration codes, safety_*, moderation_*), auth errors and local pre-request errors stay fatal; an explicit structured policy code wins over overflow-looking message text and stays fatal, while only the provider's own error code (error.code, the root code, or the upstream code the LiteLLM/Altimate Base gateway wraps under error.provider_specific_fields.error.code; never an arbitrary nested diagnostic field) or HTTP 413 classifies a context overflow, so a provider 413 or content_length_exceeded still compacts and a policy_violation wrapped around overflow text does not; a recoverable echo or compaction-summariser failure after a commit never persists a transient fatal error or session.error; the committed text is persisted before its message is marked finished; a recovered answer reuses a text part only when that part was buffered (never streamed), otherwise every streamed part is removed and the answer is published under a fresh part id; when the message that would carry it is a completed, error-free compaction summary (a request was queued during compaction) the summary stays intact as the compaction boundary and the answer is published on a new assistant message that is persisted before the queued request (one millisecond before its creation time) so later history still shows it as the reply to the original request; and when a request was queued during a post-commit compaction whose summariser then failed recoverably, the generation does not exit after recovering the earlier answer but re-reads persisted order and processes the queued request with tools, resolving waiting callers only with that request's own reply, the run --output collector drops removed parts from the reply and its fallbacks, and ACP (which cannot retract chunks) no longer forwards compaction-summary text as agent chunks, letting the committed delta through exactly once when the summary flag is cleared; a request that arrives during the echo still gets the earlier request's committed answer first and is then processed with tools; "newer request" is decided by the persisted creation time and insertion sequence, never by comparing caller-supplied message ids; an echo that itself triggers reactive compaction is finalized from the committed text instead of retried; an error-free other finish also delivers the committed text but keeps other as its finish reason, and an unmapped (unknown) finish delivers it and ends as stop; aborted and content-filter echoes keep their text and finish reason; the echo instruction never carries a copy of the committed output, so a compacted context cannot overflow on it; run --max-turns N tracks a completed answer_commit for the active request from the stream (no message fetch on the step-start path) and exempts its echo, including a retry after a reactive overflow, until any new real user request or a failed later commit attempt clears it; real request persistence emits session.request.started so a request from another attached client also clears it; the complete rendered text is also kept in the tool result's metadata, which the per-model tool-output cap does not touch, so a long answer survives even where the model-facing output is capped; (2) the committed text is preserved verbatim through automatic compaction and provider retries, with continuation ids tracked so a newer request is not overwritten by an older echo; (3) requests that supply an output JSON schema never see answer_commit: the tool is removed when StructuredOutput is injected, echo mode is never entered for them, and earlier commit tool parts are passed to the model as masked text so a historical stub cannot re-advertise the tool; a text question that merely mentions a JSON schema is unaffected. Loop termination, --output collection and tracing are otherwise unchanged.

Also in this PR: a repository-wide test (tool-schema-anthropic.test.ts) that validates every registered tool's JSON schema for the analyst and builder agents against JSON Schema draft 2020-12 plus Anthropic's restrictions, after a live probe showed Bedrock rejecting the first version of this tool's schema (z.tuple([]) emits prefixItems: []); the provider transform now omits the root $schema for Anthropic-family SDKs. ajv is added as a dev dependency for that test.

Base. Stacked on feat/analyst-harness-dab-2026-10 (#1441); the diff shown here is only this feature.

Behaviour deltas (was → now). Analyst text requests: final message produced by a prompt paragraph → produced by calling answer_commit and echoing its text on a tool-free next step (fallback to the old paragraph when the tool is off/unavailable); at the step limit, summary request → echo of the committed answer; compaction may rewrite the final text → committed text preserved. Analyst JSON-schema requests: answer_commit offered alongside StructuredOutput → not offered. Payloads: any JSON number accepted → more than 15 significant digits rejected; any size → rendered text over 60,000 characters rejected. Tables: always Markdown → Markdown, CSV, TSV or plain lines as requested. ACP clients: compaction-summary text streamed as agent chunks → not forwarded (all agents; the summary is internal compaction machinery and a later recovery may replace it). Other agents: otherwise unchanged. Registry: one analyst-gated candidate added, its id reserved against external tools. Permissions: one analyst allow entry. Anthropic-family requests: root $schema no longer sent.

How did you verify your code works?

Unit/integration, run outside any sandbox, after an independent adversarial review (approve-with-changes) and an Engineer independent review, with every bot P0/P1 through three rounds fixed:

  • test/altimate/tools/answer-commit.test.ts: renderer goldens for every shape, validation errors (empty value, ragged rows, two-of-three shapes, missing basis incl. answer-only, basis over 400 characters, precision outside 0 to 10 naming the range, parsed precision-loss literals, rendered-size cap at 59,999 / 60,000 / over with escaped cells and basis counted), filler tolerance, boolean-string coercion, label dedupe, per-column row precision (integers to fixed decimals, strings untouched, independent per column), row formats (CSV with a comma and a quote in one cell, TSV with a tab, lines, precision in CSV, Markdown default, size cap after quoting), pipe/backslash escaping, empty-tuple regression, tool-description contract incl. the one-field-per-requested-item wording on the description and the fields schema.
  • test/agent/answer-commit.test.ts: analyst sees the tool; builder does not; tool_lookup does not list it; toggle off removes it; custom analyst agent receives it; real-resolver plugin precedence; MCP/plugin/file id reservation incl. Windows case; echo right after a successful commit without a step limit and with a sibling tool; failed commit and success-then-failed-retry keep tools; new user message resets, including one persisted after the snapshot or during the echo; length-limited or mismatched echo replaced with the stored text and delivered exactly once to the run output collector and ACP deltas; stop echo without text yields a completed timed part; error-free other finish delivers the committed text once and keeps its finish; non-retryable echo transport failure or two empty compaction summaries publish the committed output once and record the cause in metadata (the no-commit empty-summary control keeps its error), while aborted, content-filtered and HTTP-policy-rejected echoes keep their fatal error; commit under a 100-token tool-output cap still yields the full table; step-limit echo; compaction/retry preservation with no full-answer copy in the request; JSON-schema requests omit the tool, never echo, and keep masked commit history; a streamed partial compaction summary followed by a 503, and two empty summaries after streamed text, each deliver exactly the committed text to the real ACP handler and the run collector on a fresh part id with every streamed part removed; collector removal cases for the reply, previous reply and fallback; a 503 guardrail denial with overflow text during the echo stays fatal without publishing or compacting; a queued user after successful compaction keeps the summary, receives the committed answer on a new message ordered before it, and is then processed with tools; 23 fromError cases for policy-code versus overflow-code precedence, including a nested diagnostic overflow code that must not mask a top-level denial and a gateway-wrapped upstream overflow code that must still compact; a user queued during a 503 compaction failure (and during two empty summaries) gets tools and its own reply after the original committed answer, with run/ACP observers seeing each text once.
  • test/altimate/tools/tool-schema-anthropic.test.ts: all registered tool schemas validate (analyst and builder, with and without search tools).
  • test/session/prompt.test.ts (structured-output case updated to assert answer_commit is absent), test/mcp/lifecycle.test.ts, prompt-contract assertion in test/altimate/sql-validation-e2e.test.ts.
  • test/cli/run/budget-finalization.test.ts: --max-turns 1 run publishes the committed output on its echo turn; a run without a commit still aborts at N+1. Help snapshot updated for the option text.

Results at the current head: 146 pass / 0 fail across test/session/message-v2.test.ts and the answer_commit agent suite, 39 pass / 0 fail for the run budget subprocess suite, and 182 pass / 0 fail across the agent suite, test/session/prompt.test.ts and the run budget tests one commit earlier; the ACP and run-output suites ran green (204 pass) at the head that introduced them; the previous head ran 335 pass / 0 fail on the answer_commit tool suite, the Anthropic schema test and test/altimate/sql-validation-e2e.test.ts (prompt contract); the previous head ran 377 pass / 0 fail with test/session/prompt.test.ts, the run budget tests and the help snapshots included (the whole test/session directory last ran green at 1,425 pass two commits earlier); bun run typecheck clean; every edit in upstream-shared files (tool/registry.ts, agent/agent.ts, session/prompt.ts, session/processor.ts, session/compaction.ts, mcp/index.ts, provider/transform.ts) inside altimate_change markers.

Live model evidence (Bedrock, Claude Opus 5.5, analyst agent, this branch): the first live session on the round-3 build committed successfully and then failed on the tool-free echo request with HTTP 400 "conversation must end with a user message", because the echo instruction was appended as an assistant turn; the instruction is now the final user turn, and the repeated live session committed once with no validation error, stopped normally, and produced a final text byte-identical to the tool's rendered output (9.4 s). The mocked-provider tests passed on both builds, so the agent test now also asserts the echo turn's role. Earlier, 65 live sessions on the previous build called the tool exactly once each with zero validation errors. Not run in CI: any live model session (no network).

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Touches core session processing, compaction, run budgets, and streaming/ACP delivery paths; scope is mostly analyst text finalization but regressions could affect multi-client runs or compaction ordering.

Overview
Introduces answer_commit, an analyst-only terminal tool that validates structured answer payloads (fields, tabular rows with format/precision, or unresolved) and returns deterministic rendered text, with the analyst prompt and permissions updated to prefer it over free-form **Answer:** prose (JSON-schema runs still use StructuredOutput).

The session prompt loop then runs a tool-free echo turn that publishes the committed text once (buffered streaming, recovery from transient provider errors, preservation through compaction/budget aborts, and assistant metadata for echo diagnostics). run gains request ownership, a max-turns exemption for the echo, harness deadline abort signaling, and --output cleanup when parts are removed. ACP stops forwarding provisional compaction-summary chunks. answer_commit is reserved in the tool registry and MCP catalog; Anthropic-family tool schemas drop root $schema, with ajv-backed schema compliance tests added.

Reviewed by Cursor Bugbot for commit d8cf20806b61ec27bf07c11d25f02e73789f909a. Bugbot is set up for automated code reviews on this repo. Configure here.