feat(agent): data-grounding rule in the agent system prompt (never fabricate)

Merged
SW
4mo ago
+16-4

Part of recoupable/chat#1833 — the PRIMARY item (single biggest lever). Base: main.

Why

Across every hallucinated task email, the root cause is the agent's decision rule "no data ⇒ invent plausible data" — stated verbatim on the Apache→OneRPM run (chat c38d62cd, ord 30): "Since the API doesn't have direct CPM metrics, I'll generate a professional report with sample YouTube analytics data." It's the only failure common to all of them, and the only fix that caps hallucinated data at ~0 regardless of the data-access bugs (some metrics — e.g. YouTube CPM — have no connector, so no access fix can conjure them).

Fix

buildAgentSystemPrompt (used by runAgentStep) now always emits a DATA_GROUNDING_SECTION, first:

State only figures you retrieved from a successful tool call this run. If a data call fails/returns empty/isn't connected, say so and omit the metric (shorter honest report, or stop) — never estimate, use "industry averages", or sample/placeholder numbers.

Tests (TDD)

  • RED→GREEN: buildAgentSystemPrompt always includes the no-fabrication rule (even with empty options); updated the exact-output tests.
  • lib/chat + app/lib/workflows suites green (568); tsc + eslint clean.

Accepted tradeoff: until the enabler PRs land, some reports get thinner ("no data") — accurate-but-thin beats confident-but-fabricated. The enablers (#733 skill install, #734 artists id, #735 socials id, LinkedIn, persistence) restore real data.

🤖 Generated with Claude Code

Summary by cubic

Adds a data-grounding rule to the agent system prompt so the agent never fabricates metrics or facts. The prompt now always starts with a “never fabricate” instruction that allows only data from successful tool calls; otherwise, say “no data.” Addresses recoupable/chat#1833.

  • New Features
    • buildAgentSystemPrompt always prepends a no-fabrication data-grounding section.
    • Instructs agents to omit metrics when data calls fail/are empty/not connected; no estimates, “industry averages,” samples, or placeholders.