feat(dbt_lineage): return the dbt integration's model lineage as structured content

Merged
RA
1d ago
+3542-85

Issue for this PR

Closes #1636

Type of change

  • New feature

What does this PR do?

This branch also carries #1635 (structured lineage and column reach for impact_analysis, include_graph), merged into it; merging this PR ships both.

dbt_lineage asks the dbt adapter for the model's lineage (getModelLineage, new in the next @altimateai/dbt-integration) and returns it as structuredContent: nodes with their columns and column edges across hops (3 each way). The text summarizes it.

  • mcp serve passes metadata.structuredContent through on non-error results.
  • Read-only dbt_lineage calls (MCP or tool run without --allow-write, and plan-mode calls) never start or use the dbt adapter, which runs dbt deps, writes target/ and can run macro SQL. dbt_lineage reads the existing forceReadOnly flag (or the plan agent) before the adapter, then answers from the manifest as main does, with no note.
  • The session's own project answers. When manifest_path is left out and the session's directory is in another dbt project than the adapter's (a dbt.json saved for another project, or a server started in one), that project's manifest answers and the adapter is skipped with a reason. A call cancelled at any point before the lookup starts nothing.
  • Without a project, an adapter that starts within 120 s, or getModelLineage, the tool answers from the manifest and says why. That answer is main's, unchanged. Its structured fields, include_graph and column reach are #1635's.
  • A cancelled call stops. A result that isn't version-1 lineage (every declared field is checked, node depth within the 3 hops asked for, finite stats) takes the manifest answer.
  • The adapter is disposed on shutdown. sql_execute routing is unchanged; bridge-first is gone.

Stacked on #1609; merges after it, needing no dbt-integration release or #1368. Read-only and manifest answers never touch dbt-integration. With writes allowed it uses getModelLineage when installed (0.3.17, through #1368), else answers from the manifest: on the locked 0.2.14, 107 ms then 14 ms, 0 stray stdout lines, 0 files written. The bridge's stdout fix is #1368's gate, where the adapter first starts under Bun.

How did you verify your code works?

  • dbt-lineage-adapter.test.ts (new) and dbt-lineage-structured.test.ts, 24 tests: read-only (no adapter, even a running one; a read-only MCP sql_execute runs nothing through dbt), the adapter path, fallbacks, cancels, the deadline, the shape check, disposal. Typecheck and Marker Guard pass.
  • Review fixes each carry a test that fails without them: plan mode, adapter-start failure note, malformed results, quoted column names ("a""b", `ab``,[a]]b]`), the session's own project (write, read-only, subfolder), disposal when closing warehouses fails, the 300-model cut.
  • Targeted suites 371/371 with ALTIMATE_MCP_E2E=1 (dbt_lineage, column reach, impact_analysis, MCP server, tool CLI, per-project registry); typecheck clean.
  • git merge-tree against #1635's head is clean.
  • E2E over stdio, jaffle-shop on DuckDB, no dbt.json:
runstray stdoutfiles writtenresult
read-only00manifest, no note, 43 ms
--allow-write, local 0.3.16 build0target/dbt-integration: 8 nodes, 22 edges

Cost side effects

With writes allowed, it compiles the models in the lineage (3 hops each way) and reads their column metadata. A read-only call reads the manifest only. No model calls.

Follow-ups (not in this PR)

  • dbt-integration bump: getModelLineage needs the next @altimateai/dbt-integration; on the locked 0.2.14 a write-enabled call starts the adapter, then answers from the manifest. Guarding on the method before the start goes away with the bump.
  • Plan mode → subagent: plan mode can hand work to a write-capable subagent, which can already edit files and run SQL writes on main; propagating plan's restrictions to children is a plan-mode design gap.
  • validators/e2e-real-dbt-*.test.ts (on main) run altimate-dbt init without isolating HOME and overwrite ~/.altimate-code/dbt.json with a temp dir that is then deleted.
  • ALTIMATE_MCP_E2E: server.test.ts says CI runs these tests in a dedicated step, but no workflow sets the variable.
  • #1654: a read-only sql_execute with workspace precedence and no warehouse can start the dbt adapter (resolveDefaultTarget → ensureDbtAdapter()). Same on main; this PR doesn't touch that path.
  • column_reach counts an implicit output alias (select 1 revenue) as a possible read; it is reported as uncertain, never dropped.
  • tool CLI and daemon don't dispose the dbt adapter (same as sql_execute there today).

Screenshots / recordings

None (no UI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

https://claude.ai/code/session_01VEmKQssMKzdJnQoPnWoCBf

Summary by cubic

Changes dbt_lineage to return the dbt integration's model lineage as structured content, with the text as a compact summary; mcp serve now relays structuredContent for non-error results. dbt_lineage also takes a project-relative SQL path or a snapshot, and include_graph returns the model's lineage graph from the manifest. impact_analysis with a column traces each downstream model's compiled SQL and reports column reach.

Notes

  • Falls back to the manifest answer and states why when the adapter, its project, or getModelLineage is unavailable; the manifest answer is mapped to the same version-1 lineage shape, and an adapter result that fails the shape check (node depth beyond the 3-hop bound, non-finite stats) takes it too.
  • Starts the dbt adapter in read-only mode for lineage (compiles and reads column metadata, runs no user SQL); projects found without a dbt.json try the Python bridge first, then the configured mode.
  • A read-only call never starts or uses the dbt adapter, and its manifest answer carries no note; a session inside another dbt project than the adapter's answers from its own manifest and says so when the adapter isn't set up for it.
  • Structured results cap at 100 KB and stay marked truncated; the MCP server declares output schemas for both tools.
  • The adapter is disposed on shutdown; a cancel stops the call even mid project-read or manifest fallback.
  • Depends on the next @altimateai/dbt-integration release.

Summary by CodeRabbit

  • New Features

    • Expanded dbt lineage lookups to support model names, unique IDs, and project-relative SQL paths, with optional graph and column-level details.
    • Added column-reach tracing across downstream models and related exposures, identifying uncertain, untraced, and truncated results.
    • Enhanced impact analysis with exposure and column-level insights, unique-ID matching, and shortest downstream dependency depths.
    • Lineage tools now provide structured results alongside readable reports, with manifest-based results available when adapter lineage cannot be used.
    • Added schema-listing support.

  • Bug Fixes

    • Improved adapter cleanup after initialization failures and during shutdown, including when other cleanup operations fail.
    • Structured lineage responses now observe output size limits and indicate when results are truncated.