PR #1635 - AltimateAI/altimate-code

Issue for this PR

Stacked on #1637 (the dbt integration's structured dbt_lineage), so it merges after #1637. It no longer depends on #1622. The Claude Code plugin's lineage pane consumes this (AltimateAI/altimate-claude-plugin#14).

Type of change

Bug fix
New feature
Refactor / code improvement
Documentation

What does this PR do?

Gives clients that draw lineage structured data instead of text, and makes column impact trace real downstream SQL. On top of #1637: when the dbt integration answers, dbt_lineage keeps its model lineage as the structured result, adding the manifest graph on include_graph; the manifest fallback reports its own lineage fields.

dbt_lineage reports the model's upstream ids, column mappings and edges in its metadata. It finds a model by name, unique_id or project-relative SQL path, and a snapshot by name or id. With include_graph it also returns the model's graph: everything upstream and downstream, up to 500 nodes each side (upTruncated/truncated when a side is cut). This is for clients that cannot read a manifest over 4 MiB; over MCP the graph is left out of the structured result, so they read it through altimate tool run.
impact_analysis with a column now traces each downstream model's (and snapshot's) compiled SQL in dependency order, so renames are followed. It returns column_reach:
- certain readers: the model and the output columns the traced column feeds;
- uncertain: models that name the column outside the select list, select * from a parent whose schema.yml does not list it, use a NATURAL JOIN or name it through Snowflake's IDENTIFIER('…'), read a parent whose table name another parent shares, read an ephemeral parent, or sit downstream of one of these;
- untraced: no compiled SQL, or SQL the engine could not parse;
- exposures, split into those on certain readers and uncertain_exposures.
It never reports a confident zero for what the engine reports it cannot settle. One known gap: a parent's whole row passed as a value (to_json(o)) gives no edge and is not caught (see Follow-ups). It also reports downstream, exposures, and column_reach_error. The model can be given by unique_id, and downstream matches by exact id when one is given (versioned models, same names across packages). The dialect override is honoured; without it, the dialect is detected from the manifest.
MCP server: both tools declare an output schema and return those fields as structured content, with the text report as report. For dbt_lineage, report is only the header, since the mappings travel as data. When the dbt integration answers, its model lineage is the structured result. The structured content is capped at 60 KB, measured as serialized JSON, so it stays under Claude Code's 25k-token MCP output limit; the dbt integration's report is sent as its header only, since its edges travel as data. Above that only the report is sent, marked truncated, and the report itself is shortened if it is still too large.
Adapter and manifest from the same project: when manifest_path names a manifest outside the dbt integration's project (another folder, or a nested project inside it), dbt_lineage uses that manifest and says why, so one project's lineage is never mixed with another's graph. A manifest not written yet (an uncompiled project) is placed by its path, so the project's own target/manifest.json keeps the integration's answer.

How did you verify your code works?

bun test test/altimate/column-reach.test.ts test/altimate/dbt-lineage-structured.test.ts test/altimate/tools/impact-analysis.test.ts test/altimate/impact-analysis.test.ts test/altimate/dbt-lineage-adapter.test.ts: 106 pass (with dbt-lineage-helpers.test.ts, 110 across the six files). The tests cover:
- renames, filters, wildcard forms;
- ambiguous and qualified relations, quoted dotted and case-distinct columns;
- snapshots, ephemeral parents, uncompiled and unparseable SQL;
- versioned and cross-package ids, report sections, structured-result cap;
- the adapter path's model lineage and graph, a manifest from another or a nested project, and an uncompiled project's own manifest path;
- NATURAL JOINs, IDENTIFIER() reads, a manifest path that is a link or runs through a file, and SQL paths spelled with ./ or backslashes;
- include_graph returning a twelve-hop chain whole to its source, and the 500-node cut on each side;
- review follow-ups: a negated operand is not select *, column spellings that differ only by case are reported as uncertain, an empty column_lineage falls back to column_dict, dollar-quoted literals are blanked, graph_error explains a missing graph, a model the engine traces only partly or finds no edges in is uncertain, never a confident non-reader, and dashboards on a metric are found through its semantic model.
ALTIMATE_MCP_E2E=1 bun test test/altimate/mcp-server/: 115 of 116 pass, including the structured output. The one failure, warehouse_list discovery stops starting calls once its deadline passes, comes from the base branch. It timed out in the full 16-minute run but passes alone at this PR's head and on the base branch, and this PR does not touch warehouse_list.
bun run typecheck: clean. The marker guard and tracker-leak check pass.
Ran against a compiled dbt-duckdb demo project (20 models, 3 dashboard exposures) through altimate tool run and the MCP server inside Claude Code. fct_orders.revenue is read by 7 downstream models and 3 dashboards, which matches the SQL by hand.
Column reach over 300 downstream models of 41 columns each takes 548 ms.
Not from this PR: when these test files run together in one process, 41 MCP E2E tests and 2 runner tests fail. They fail the same way on the base branch and pass file by file.
Reviewed by Claude and Codex as separate reviewers, each run on its own, until both explicitly approved.

All tests

Every test this PR adds (57). It also extends existing tests: the adapter-project test in dbt-lineage-structured.test.ts and the structured-output checks in mcp-server/server.test.ts.

column-reach.test.ts: 45 tests

columnReach
- follows a column through renames to the models and dashboards that read it
- an output alias or a comment naming the column is not a read
- a downstream model without compiled SQL is reported untraced, and what reads it only may read the column
- a model that only filters on the column passes nothing on to its children
- a join key another parent also documents is a possible read, not a dropped one
- a source and a model with the same table name: a qualified source column is not the model's
- a table name two parents share is ambiguous; a schema-qualified one is not
- a snapshot is traced and carries the column on
- a column no downstream model reads reaches nothing
- an unknown model or a missing manifest throws
exposuresOn through metrics
- an exposure on a metric reaches the model under its semantic model
exposuresOn
- lists exposures on any of the given nodes, by label when there is one
parentsNamed
- a qualified name matches the parent at that qualification only; a bare name matches every parent that answers to it
columnReach where the engine's schema falls short
- select * from a parent whose schema.yml does not list the column is a possible read, and its children may read it
- a dashboard downstream of a wildcard reader in any dialect form is reported, not dropped
- a column name with a dot, quoted, is matched whole
selectsStar
- a CR-terminated line comment does not hide the SQL after it
- finds * and t.* in a select list, not multiplication or count(*)
columnReach through an ephemeral parent
- a model reading an ephemeral parent is a possible read, never a silent zero
columnReach with an ephemeral and a regular parent
- a traced edge from the regular parent does not hide what flows through the ephemeral one
columnReach with columns that differ only by case
- "Revenue" is not revenue where the parent documents both
columnReach case handling
- a spelling matching one documented column ignoring case reaches its readers
- a spelling matching two columns that differ only by case leaves both columns' readers uncertain, never certain
- with the spelling unsettled, a dashboard on the model itself is uncertain too
columnReach with an empty column_lineage
- falls back to column_dict edges
columnReach on a model the engine cannot fully trace
- no edge at all: SQL that names the column (a PIVOT) is uncertain and so is its child; SQL that never names it reads nothing
- errors beside edges: a reader stays reached but unsure, a non-reader is uncertain
columnReach on SQL the engine cannot parse
- is untraced, and a dashboard on it is reported apart
lineageGraph
- a twelve-hop chain comes back whole to its source; upstream past 500 nodes is cut and marked
- downstream past 500 nodes is cut and marked; downstream tests are still counted
- everything up and downstream, tests as ids, from dbt's child_map when present
- a node whose manifest relation_name is null has no relation
structuredError
- passes a short message and caps a huge one
- a tool's own structured result sends the report's header, not its edge lists
- exposures_error reaches the structured fields
structuredResult over the cap
- a report too big on its own is shortened too
structuredError escaping
- the cap holds for the serialized result, whatever JSON escaping adds
structuredResult
- passes the lineage fields with the report, and only the report when they are too big
dbtLineage
- reports the model's upstream unique_ids, and the graph when asked
columnReach with a NATURAL JOIN
- a NATURAL JOIN is a possible read, never a confident non-reader
- a model that reads the column through a NATURAL JOIN passes the doubt on to its children
- every NATURAL join form counts, whatever the quoting around it; aggregates under their own names do not
- a NATURAL JOIN model that names the column outside its select list also passes the doubt on
columnReach through Snowflake IDENTIFIER()
- a column named as IDENTIFIER('…'), plain or escaped, is a possible read, and its child and dashboard are not dropped
- IDENTIFIER() with the column, quoted or qualified, or a bind variable counts; another column does not

dbt-lineage-helpers.test.ts: 1 test

dbtLineage: error handling
- reports the model's name, not the path used to find it, when there is no compiled SQL

dbt-lineage-structured.test.ts: 2 tests

dbt_lineage with the project's dbt adapter
- a manifest named outside the adapter's project wins over the adapter
- include_graph with no manifest to read says so in graph_error

impact-analysis.test.ts: 1 test

impact_analysis: DAG traversal
- the structured downstream entries carry no path

tools/impact-analysis.test.ts: 8 tests

findDownstream: depth
- a direct dependent also reachable through another dependent stays direct
formatImpactReport: exposures
- says when exposures could not be read
- keeps a multi-line read error on one line
- lists a dashboard once when column reach already printed it
formatImpactReport
- column-level impact names the column and the change
- column reach lists certain readers, possible readers, untraced models and exposures
findDownstream with versioned models
- a versioned parent is matched by unique_id, whose last segment is the version
- with an exact id, another package's model of the same name is not matched

Follow-ups (not fixed here):

On the adapter path, a model the adapter knows but an older target/manifest.json does not gets a graph node with no edges. dbt parse fixes it.
impact_analysis's downstream traversal (from #350) scans every model per level (O(V·M)). An adjacency map would make it O(V + E). It also walks models only, so a snapshot and what sits below it are missing from downstream, the severity count and exposures, while column_reach traces through snapshots. Walking the manifest's child_map over models and snapshots fixes both. This is pre-existing and unchanged here.
Whole-row reads (to_json(o), md5(cast(o as varchar))): the engine reports no column edge, and a pattern can't tell a row source from a column (sum(x) as x, extract(year from d)), so these models are confident non-readers today. The engine reporting a whole-row reference would settle it.
detectDialect (pre-existing) passes synapse and fabric through, which the engine rejects, so their models come back untraced (reported, never a false zero). Mapping both to tsql would trace them.
Overlap with #1637: once dbt-integration ships getModelLineage, impact_analysis's column reach can ask it first (its column edges across hops) and keep this PR's compiled-SQL tracer as the fallback for projects without the dbt integration. Today the tracer is the only path that works, since the integration is not available yet.

Cost side effects

None. No model calls or warehouse queries. Column reach parses compiled SQL locally, at most 300 models per call.

Screenshots / recordings

The plugin PR has screenshots of the pane that uses this data. A demo recording will follow.

Checklist

I have tested my changes locally
I have not included unrelated changes in this PR
I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code