refactor: split builder prompt into invariant core + named packs (byte-identical assembly) + opt-in data-qa profile

Merged
AN
2mo ago
+648-240

Issue for this PR

Closes #1216

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Compile-time split of the monolithic builder prompt into an invariant core plus named packs, with the assembled default output byte-for-byte identical to the pre-split builder.txt. That identity is the entire quality argument for the default path: identical bytes cannot change behavior, so no eval run is needed to prove non-regression.

Structure (packages/opencode/src/altimate/prompts/):

  • builder/core.txt — identity + principles 1–3 (workload-independent)
  • builder/core-training.txt — Teammate Training section
  • builder/packs/dbt-ops.txt — principle 4 (full build), tool access list, dbt Operations
  • builder/packs/sql-guard.txt — Pre-Execution Protocol
  • builder/packs/dbt-verify.txt — dbt Verification Workflow
  • builder/packs/dbt-workflow.txt — Workflow
  • builder/packs/pitfalls.txt — Common Pitfalls
  • builder/packs/self-review.txt — Self-Review Before Completion
  • builder/packs/legacy-skills-catalogue.txt — Skills catalogue + Proactive Skill Invocation (kept, per the design: it only gets deleted once the sibling measurement lands)
  • builder/packs/finish.txt — Finish Protocol
  • profiles.ts — assembles profiles by plain join("") of fragments (each fragment carries its own trailing newlines), at module load via the same Bun .txt import mechanism the single file used. agent.ts now imports PROMPT_BUILDER from here; the prompt flows into sessions unchanged (llm.ts uses input.agent.prompt verbatim; the old file had no template placeholders, and none were introduced).

Not a single character of prompt text was edited — this PR moves text, it does not improve it. (Tempting wording fixes noted for a later, separately-measured PR: the catalogue/Skill.fmt double-advertisement, and the builder mode identity line being dbt-flavored for non-dbt profiles.)

Opt-in data-qa profile: the builder prompt minus the dbt-specific packs and the Pre-Execution Protocol pack, with identical tool permissions to builder. Registered only when ALTIMATE_DATA_QA_PROFILE=1 is set or config defines agent.data-qa, and even then never auto-selected — the default agent remains builder; a user must pick it explicitly (--agent data-qa, TUI agent cycle, or config). Basis: an internal 540-trial paired prompt ablation on a public benchmark found removing these sections on data-Q&A workloads had no score effect (permutation p=0.74) and cut wall clock 27.6%.

On the size of the win — read this before citing a percentage. The assembled data-qa prompt is 7,163 bytes vs 14,773 (a −51.5% cut to the system-prompt text). That number is smaller than it looks at the request level: a measured opening request on a data-Q&A workload was ~34,600 tokens, of which the system prompt is ~18% and tool schemas are ~82% (92 tool definitions, ~28K tokens). So trimming the prompt moves ~5.4% of the actual payload, not half of it. The durable value of this PR is therefore the composition mechanism — a profile can declare what a workload needs — not the token saving. The larger lever on request size is tool-schema selection, which is a separate follow-up and out of scope here. As shipped, data-qa is dormant (nothing selects it), so it changes nothing until deliberately enabled and is best read as the proof that the assembly mechanism composes into more than the monolith.

Deviations from the design doc, and why

  • The doc sketches data-qa as core + a new pack harvested from analyst.txt. This PR builds it subtractively (builder minus packs) instead: PR 1's contract is "moves text, authors none", and harvesting new prose is a content change that deserves its own measured PR.
  • Two fragments beyond the doc's named set (dbt-workflow, and core split into core + core-training) because the core sections are non-contiguous in the original file and the split must be mechanical.

Upstream note: builder.txt was 100% fork-grown (created in the fork's first rebrand commit; no upstream ancestor), so the split touches no upstream-shared prompt content. The loader (agent/agent.ts) is upstream-shared; all edits there sit inside altimate_change markers and the strict marker check passes.

How did you verify your code works?

In order of strength:

  1. Byte-identity gate (test/altimate/prompt-profiles.test.ts): the assembled default profile must hash to the pinned sha256 of the pre-split builder.txt (17663410dd9a…, 14,773 bytes) and be a plain ordered concatenation of the fragments with full coverage. This is the load-bearing proof.
  2. Negative control for the gate itself: appended one byte to packs/sql-guard.txt → the identity test fails (Expected: 17663410… Received: a1f8e155…, 4 tests fail); reverted → 7 pass. The gate can fail, so its green means something.
  3. Assembly determinism: two fresh bun subprocesses, different cwd and different HOME, both reproduce the pinned hash + byte count.
  4. Registry-level tests (test/agent/data-qa-profile.test.ts, real Agent service): with no flag, data-qa does not exist and builder.prompt carries the pinned bytes; with the flag, data-qa exists with the expected composition, builder is unchanged, and defaultAgent() still resolves to builder.
  5. End-to-end through the product path — layer exercised: the real CLI entry (src/index.ts → instance boot → config load → real Agent service), i.e. the same object session/llm.ts reads input.agent.prompt from; not the unit-test layer:
    • debug agent builder from a scratch directory → served prompt hashes to the pinned sha256, 14,773 bytes.
    • debug agent data-qa without the flag → exit 1, "Agent data-qa not found".
    • ALTIMATE_DATA_QA_PROFILE=1 … debug agent data-qa → exit 0; served prompt omits Pre-Execution Protocol / dbt Operations / Finish Protocol and retains Skills + Teammate Training.
    • Not verified end-to-end: a live model round-trip (no LLM call was made); byte-level identity of the served prompt makes that redundant for the default path.
  6. Gates: bun run typecheck clean; bun run script/upstream/analyze.ts --markers --base origin/main --strict clean; oxlint — zero findings in changed files (the repo-wide run reports one pre-existing error, a bun-types tsconfig resolution failure under sdks/vscode, untouched here); full packages/opencode test suite run with pre-existing failures separated (details in first PR comment).

test/session/termination.test.ts read the deleted builder.txt from disk; it now asserts the same thing against the assembled PROMPT_BUILDER.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

[!NOTE]
Medium Risk
Changes default agent system-prompt delivery and adds a new native agent path; behavior is guarded by byte-identity tests but any future fragment edit without updating the pin could silently change builder prompts.

Overview
Replaces the monolithic builder.txt with compile-time assembly in profiles.ts: invariant core plus named pack fragments concatenated at module load. The builder agent now uses PromptProfiles.PROMPT_BUILDER, which must stay byte-for-byte identical to the old file (pinned sha256 + subprocess determinism tests). .gitattributes forces LF on prompt paths so autocrlf cannot break that pin.

Adds an opt-in data-qa primary agent with the same tool permissions as builder but a slimmer prompt (core, skills catalogue, teammate training only—no sql-guard / dbt build packs). It registers only when ALTIMATE_DATA_QA_PROFILE, agent.data-qa in config, or default_agent: "data-qa"; otherwise the default stays builder. Run-mode DONE completion injection now applies to data-qa as well as builder in termination.ts.

Reviewed by Cursor Bugbot for commit 24bbab85e166fb97624272c055ab2d9e337ec859. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features

    • Added an optional data-QA agent profile for specialized data-quality workflows.
    • The data-QA profile can be enabled through an environment setting or agent configuration.
    • Added expanded guidance for dbt development, SQL safety, validation, self-review, and completion workflows.
    • Data-QA runs now receive the same completion instructions as builder runs.

  • Bug Fixes

    • Data-QA can now be selected as the default agent when explicitly configured.
    • Preserved the existing builder prompt behavior while introducing profile-based prompt composition.