fix: dbt_cte_profile lists the earlier CTEs each CTE reads
Issue for this PR
Closes #1794
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
dbt_cte_profile lists a model's CTEs and times each one with a count query over that CTE and the CTEs before it in the same WITH clause. The Claude Code plugin's /profile-ctes worked out each CTE's own time as its count's time less the CTE before it. Warehouses skip the CTEs a count doesn't use, so a CTE that reads none of the earlier ones (a sibling) was really timed on its own. The subtraction then gave 0 ms whenever the CTE before it was slower. In the demo shop's customer_revenue, payments (4.4 s on its own) showed as +0 ms after order_items (15 s).
The listing now gives each CTE reads: the positions of the earlier CTEs of its own WITH clause that its body names. The text output adds "reads a, b" to each line. A client can then take a CTE's own time as its count's time less the own times of the CTEs it reads, directly or through another. The plugin does this in a companion change.
cteReads in cte-detect.ts reads each CTE body in two passes.
- Jinja is blanked first, because it renders before the SQL is read. Every
{{ }}(brace-balanced, quote-aware),{% %}and{# #}becomes spaces, and so does the body of a capturedset,callormacroblock, which is a string for dbt. Offsets are kept. Text inside{% raw %}is kept as written. - The SQL scan then skips strings (with doubled and backslash-escaped quotes), dollar quotes and comments. It reads
"…",`…`and[…]names whole. A name after a dot, even across a comment, is not a reference, and neither is one inside a longer word. Names match in any letter case and quoting, because Snowflake folds bare names to upper case and most others to lower. - A name that the body's own WITH defines again shadows the outer CTE.
- Only the CTE's own WITH clause counts. A per-clause name map keeps the scan linear: 10,000 CTEs take milliseconds.
Known limits, stated in the doc comment:
- A column or alias spelled like an earlier CTE counts as a read.
- A CTE named only through Jinja isn't seen.
- Shadowing in a deeper subquery isn't seen.
- Nested block comments aren't handled.
- Where a backslash is a plain character in a string (Postgres, DuckDB), a string that ends in one pairs the later quotes wrongly.
dbt_lineage's include_graph nodes now also carry yml: the node's patch_path without its package:// prefix. With it, the plugin's lineage pane can say when a model's schema.yml changed after the manifest was written, for manifests over 4 MiB whose graph altimate-code builds. A model's test count comes from its schema.yml.
How did you verify your code works?
bun run typecheck(tsgo) passes.bun test test/altimate/cte-detect.test.ts test/altimate/dbt-cte-profile.test.ts test/altimate/column-reach.test.ts: 230 pass.- New tests, each of which fails on
main(cteReadsdoesn't exist there, and the listing has noreads):- siblings read nothing of each other, and a CTE that joins three earlier ones lists all three (the
customer_revenueshape); - a name in a string, a comment, after a dot or inside a longer word is not a read;
- quoted and bare names match as described, and a WITH inside a subquery is not mixed in;
- Jinja in strings, comments, dollar quotes and bracketed names, Jinja quotes and delimiters inside Jinja, captured blocks,
{% raw %}, nested braces, escaped quotes, names qualified across comments, non-BMP names and shadowing by a body's own WITH; - a 10,000-CTE model is listed in linear time;
- the tool's listing carries
readsand prints it; lineageGraphnodes carryymlwithout the package prefix.
- siblings read nothing of each other, and a CTE that joins three earlier ones lists all three (the
- Reviewed by gpt-6-astra and gpt-6.1-sol (high) over 11 rounds, with each round's findings fixed with a regression test. Both approved 26df3ff05b. Shadowing by a WITH (the body's own, a subquery's, or WITH RECURSIVE) and the quoted-vs-bare name rule were added along the way. A bare name matches a quoted one only as its lower- or upper-case form.
- Live:
altimate tool run dbt_cte_profile --args '{"model":"customer_revenue"}'on a copy of the demo DuckDB shop listscustomer_orders (line 33; reads orders, order_items, payments)andranked (line 47; reads customer_orders), withreads: []for the four independent CTEs.
Cost side effects
None. The listing reads the model file it already reads; no query is added.
Screenshots / recordings
Not a UI change.
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
- I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
Summary by cubic
Fixes dbt_cte_profile so CTE timing is based on what each CTE actually reads instead of assuming it reads the CTE immediately before it. A sibling that follows a slower CTE it does not read no longer shows 0 ms.
Each listed CTE now carries reads: the positions of the earlier CTEs in its own WITH clause whose names its body uses, shown by name in the text output; a client subtracts the closure of reads for a CTE's own time. Name detection blanks Jinja first ({{ }}, {% %}, {# #}, and the body dbt captures in set/call/macro blocks), since Jinja renders before the SQL is read, then scans the SQL; a CTE the body's own WITH or a subquery's WITH defines shadows an outer CTE of the same name once its definition has ended. In a WITH RECURSIVE clause, recognized across any whitespace and comments, an inner CTE shadows from the clause's start, since its name already means that CTE within its own body. Detection skips strings, comments, names after a dot even across a comment, and case-folded names; bracketed names match whole, and names may start outside the basic multilingual plane. dbt_lineage graph nodes now carry yml, the project-relative schema.yml that describes the node, so a client of a large manifest can detect when a model's schema changed.
🤖 Generated with Claude Code
Summary by CodeRabbit
- New Features
- CTE listings now show which earlier CTEs each CTE reads, including the dependency positions in metadata.
- dbt graph entries can include the associated schema file path.