feat: add text_label bulk labeling tool for the analyst, surface provider stream errors

Merged
AN
4d ago
+1371-27

Issue for this PR

Closes #1535

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why. Questions that need classifying many free-text rows (topic, intent, sentiment) had no tool, so the analyst fell back to keyword/regex guesses or read rows into the conversation. Separately, streamText fullStream error parts were discarded, so a provider 400 surfaced only as NoOutputGeneratedError.

Before / after.

  • Before: no bulk semantic labeling; stream errors in skill-selector, enhance-prompt, ai-review hidden.
  • After: text_label labels every row of a scratch query against a fixed label set (with descriptions) in bounded concurrent batches outside the model context, writes a scratch table, and reports coverage rows_in / labeled / failed / unparseable (failed ids retrievable via SQL). Optional spot_check measures repeat-label agreement (stability, not accuracy). Non-transient 4xx stops retries; transient errors retry with backoff. Unlabeled rows are never dropped: they are stored with a status and error. The data-question skill now points bulk labeling at text_label and keeps the rule against claiming labels that were not computed. The real provider error now propagates via a shared altimate/llm/stream-text.ts.

Flag. ALTIMATE_TEXT_LABEL=0 disables the tool (default on, analyst only).

Two commits: fix: for the stream-error helper (separable), feat: for the tool.

Gemini 4 live use depends on #1534 (thinkingLevel fix) and #1532 (empty-enum tool schema) merging; neither is included here.

How did you verify your code works?

  • bun test on text-label, text-label-model, stream-text, tool-schema-anthropic, enhance-prompt, skill-filtering, review, release-v0.8.1-adversarial: 240 pass, 0 fail. bun run typecheck clean.
  • Review round 2 (head 7254d081e7): text_label is now scoped to the analyst and data-explorer agents (builder does not receive it); the oversized-input error and data-question skill state the 30,000 row limit and the split-and-verify-coverage procedure; the schema test rejects any enum containing an empty string; the model-resolution test now persists the assistant message through the legacy session API so its assertions execute (verified it fails when the asserted behavior is broken). Text-label, model, schema and test/agent suites: 205 pass, 0 fail; typecheck clean.
  • Deferred follow-up (valid P2): nested labeling streams do not yet add usage to session totals.
  • Marker Guard run locally (reports no upstream-shared files on current main, so edits were audited by hand: registry and agent edits sit inside altimate_change blocks).
  • Live check (throwaway local branch with #1534 and #1532 cherry-picked, not in this PR): altimate-code run --agent analyst -m google-vertex/gemini-4-argon on 30 short texts into praise / complaint / question, spot_check: 10. Result: rows_in 30, labeled 30, failed 0, unparseable 0; SQL counts 10 / 10 / 10.
  • Stream-error live check on plain code: a bad request to Vertex. Old drain-then-.text path gave NoOutputGeneratedError: No output generated; readStreamText gave AI_APICallError: ... thinking_budget is only supported when thinking is enabled.

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
New analyst-facing tool issues many LLM calls and writes session scratch tables; misconfiguration or partial runs could mislead counts if coverage checks are skipped, though failures are persisted explicitly.

Overview
Adds a text_label tool so the analyst (and read-only data-explorer) can classify every row from a scratch SQL query against a fixed label rubric via the session model, without pulling rows into chat or using keyword heuristics. Labeling runs in bounded batches with concurrency limits, transient retries, deadlines, and strict JSON parsing; results land in a new scratch table with coverage stats (rows_in / labeled / failed / unparseable), optional spot_check agreement, and SQL helpers for failed IDs and counts. Registration is analyst/data-explorer scoped and can be turned off with ALTIMATE_TEXT_LABEL=0; permissions allow text_label on the read-only data agents.

Introduces shared readStreamText so auxiliary LLM streams surface real provider errors from fullStream instead of masking them as NoOutputGeneratedError; enhance-prompt, skill-selector, and ai-review now use it, and text_label uses it for model calls.

The data-question skill now directs large semantic labeling through text_label + sql_materialize and tightens rules against claiming labels that were not persisted.

Reviewed by Cursor Bugbot for commit 7254d081e7dd322e07b99f0bdd1fc79e03929a7a. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Adds a text_label tool that labels every row of a scratch query against a fixed label set, and surfaces the real provider error from streamText instead of a generic NoOutputGeneratedError.

New tool

  • Labels rows in bounded concurrent batches outside the conversation, writes results to a scratch table, and reports coverage (rows_in / labeled / failed / unparseable) with failed IDs retrievable via SQL.
  • 401/403/404 stop the run immediately; batch-specific 4xx fail only that batch; other permanent 4xx stop when repeated with no successful batch; transient errors retry with backoff.
  • NULL source text, oversized text, empty populations, and duplicate or invalid IDs are rejected or failed with explicit errors before labeling. Every row is persisted with a status and error—unlabeled rows are never dropped.
  • Optional spot_check measures repeat-label agreement (stability, not accuracy).
  • Enabled for analyst and data-explorer agents only; disabled globally via ALTIMATE_TEXT_LABEL=0. Capped at 30,000 rows per call, with larger populations split across disjoint ID ranges. The data-question skill now points bulk labeling at text_label.

Stream errors

  • skill-selector, enhance-prompt, and ai-review now share readStreamText, which propagates the provider's error instead of NoOutputGeneratedError.

Summary by CodeRabbit

  • New Features
    • Added AI-assisted text labeling for data in scratch tables, with configurable labels, bounded batch processing, and persisted results.
    • Labeling results report coverage, failed or unparseable rows, label counts, and optional spot-check agreement.
    • Added optional prompt enhancement that can clarify prompts while preserving their original intent.
  • Improvements
    • Labeling guidance now explains how to verify coverage and report limitations when complete annotations cannot be saved.