feat: dbt_cost_estimate on Snowflake and Databricks from the planner's EXPLAIN
Issue for this PR
No GitHub issue. Related to #906 (pre-execution cost firewall). This is the CLI side of the Claude Code plugin's scan check on Snowflake and Databricks; the plugin PR, AltimateAI/altimate-claude-plugin#39, depends on this release.
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
dbt_cost_estimate only read BigQuery's dry run. On any other warehouse it answered supported: false, so the plugin's scan check never held anything on Snowflake or Databricks. Neither has a dry run, so this PR reads the query planner's estimate from EXPLAIN, which compiles the query without running it:
| Warehouse | Statement | Figure read | Accuracy |
|---|---|---|---|
| BigQuery | dry run (unchanged) | bytes_processed | unchanged: maxBytes bound, or lowerBound for scripts and zeros |
| Snowflake | EXPLAIN USING JSON <sql> | GlobalStats.bytesAssigned (plus partitionsAssigned / partitionsTotal) | Snowflake documents it as an upper-bound estimate: the bytes in the micro-partitions left after compile-time pruning. Runtime pruning (join filters) can scan less, and a cached result scans nothing. |
| Databricks | EXPLAIN COST <sql> | the sum of Statistics(sizeInBytes=…) over the leaves of == Optimized Logical Plan == (the relations scanned) | An estimate, not a bound either way. These are table and file sizes (Delta log, catalog) before column pruning and runtime file skipping, so a run usually reads less. A table without statistics has no estimate. |
- No estimate, never a block. If a Databricks leaf has no
Statistics, or shows Spark's default for a table without statistics (spark.sql.defaultSizeInBytes, Long.MaxValue, printed as8.0 EiB; anything at or above 2^62 is treated the same), the tool answerssuccess: false. It names the relation when Spark prints a plain name (letters, digits, underscores, dots) and suggestsANALYZE TABLE <relation> COMPUTE STATISTICS NOSCAN, which records the size from file metadata without reading the data (a full ANALYZE scans the table). A failed EXPLAIN or an unreadable plan also givessuccess: falsewith the reason. None of these is read as a count. - Result fields:
source: "explain",bound: "upper"on Snowflake and"estimate"on Databricks,bytes,display(the dry run's units, e.g.2.0 GiB), andpartitionsAssigned/partitionsTotalon Snowflake. There is nomaxBytes, so a consumer comparesbytesitself. BigQuery metadata is unchanged (nosourcefield), so a consumer that sees nosourcetreats the result as a dry run. - How EXPLAIN runs: the compiled model or SQL is wrapped in inline Jinja,
{%- if execute -%}{%- set altimate_plan = run_query('EXPLAIN … <sql>') -%}{{ tojson(altimate_plan.columns[0].values() | list) }}{%- else -%}[]{%- endif -%}, and passed tounsafeCompileQuery. Theexecuteguard matters becausedbt compile --inlinerenders the query once at parse, whererun_queryreturns None; that pass renders[]and runs nothing. It runs on dbt's own connection, the same way dbt-integration'sgetColumnsOfSourcedoes. It goes through compile rather thanimmediatelyExecuteSQLbecause that path isdbt show --limit N, which wraps the query in a limit subquery that an EXPLAIN can't take. The SQL is written into a Jinja string literal, so quotes and backslashes are escaped and trailing semicolons and whitespace are dropped. - Never a script. dbt-snowflake runs every statement that the connector's
split_statementssplits off, so a second statement after the EXPLAIN would run for real.mayBeScriptreads the SQL in one pass, the way that splitter does: single- and double-quoted text where a backslash escapes, backticks on Databricks,--and/* */comments. Anything it doesn't model exactly counts as a script and gets no estimate, with nothing sent: a Databricks--comment continued past a trailing backslash,$$text,//comments, a line starting with!(a Snowflake client command, which the splitter sends on its own), Databricks raw strings, nested or unclosed comments, unclosed quotes,EXECUTE IMMEDIATE, or anything after a;but whitespace, comments or more semicolons. Line endings are normalised first, as Jinja does (\r\nand a lone\rbecome\n), so the text checked is the text the driver gets. - No EXPLAIN for queries over files on Databricks.
read_files(…),cloud_files(…), path queries over a known format (json.`…`,libsvm, …; a format word in the middle of a dotted name such asmain.json.`t`is a table; a column qualified by an alias named like a format is refused, since that can't be told apart without parsing), and any backtick-quoted name that looks like a path (contains/or:), whatever the source before it (a format, a provider class), unless it follows Delta's own provider, get no estimate, because Databricks infers their schema by reading the files while it plans. Detection uses the same single scan as the script check (comments and quoted text blanked, backtick-quoted names kept) and accepts backtick-quoted names and comments between tokens (read_files/**/(,`json`.`…`). Delta paths and tables are estimated. - Other warehouses still answer
supported: false. The message now names the three supported warehouses.
Supported input (EXPLAIN parsers)
parseSnowflakeExplain(content): thecontentcolumn ofEXPLAIN USING JSON, as a string or already parsed. ReadsGlobalStats.bytesAssigned(a finite number ≥ 0). Partition counts are optional. Anything else (the TABULAR/TEXT formats, a missing or non-numericbytesAssigned) returnsundefined.parseDatabricksExplainCost(plan): theplancolumn ofEXPLAIN COST, i.e. Spark'sTreeNode.generateTreeStringwithverboseStringWithSuffix. Each tree level is a 3-character prefix (:/for ancestors,:-/+-for the node). Inner children such as scalar subqueries sit at depth+2, and their relations count. Sizes useUtils.bytesToString:%.1fwithB, KiB … EiB, Spark 2'sKB … EB, or1.23E+21 Bpast a long. Only the== Optimized Logical Plan ==section is read, up to the next Spark section header (Parsed/Analyzed/Optimized Logical Plan,Physical Plan). CRLF is accepted.mayBeScript(sql, type)is conservative by design: a false "script" gives no estimate, never a run.- A Databricks leaf's size is read only from the
, Statistics(sizeInBytes=…[, …])suffix at the end of the node line, so a column named like it is ignored. - Documented limits:
- A node whose text spans lines (a string literal containing a newline) is joined to its node when the continuation line has no tree prefix. One that happens to start with tree-like spaces may still be read as an extra node, giving no estimate or a higher one, never a lower one.
- If
spark.sql.defaultSizeInBytesis set below 2^62, a table without statistics is read at that size. - An inner-child plan printed without statistics gives no estimate.
How did you verify your code works?
bun test test/altimate/dbt-cost-estimate.test.ts: 195 pass. New cases cover:-
parsers on documented-shape outputs: the Snowflake JSON from the EXPLAIN docs, and Spark plans built from
Statistics.simpleString,Utils.bytesToStringandgenerateTreeString(joins, subqueries, default-size and missing stats, units, CRLF, malformed input) -
the Jinja builder and its escaping
-
the tool on Snowflake and Databricks: compile, EXPLAIN, and the metadata above; no stats, a refused EXPLAIN, an unreadable plan, a failed compile, cancellation
-
from review:
- the parse-pass guard
- script refusal (
select 1; delete from ordersis never sent), including separators hidden by$$,//, raw strings, escaped double quotes, and nested or unclosed comments - statistics-like column names in Databricks plans
- a lone
\rthat Jinja turns into a line break !commands at a line's start, nested and overlapping (/*/*/) comments, quoted text after a;- relation names Spark prints unquoted with a comma, hyphen or space: the advice names only plain ones
- multi-line plan nodes, and Databricks queries over files
- stray
== … ==lines inside a plan
Each fails on the code before its fix.
-
- Real dbt, real Jinja: the generated template went through
dbt compile --inline(dbt-core 1.11, dbt-duckdb, local DuckDB, where plainEXPLAINruns).- Without the guard, dbt failed at parse:
'None' has no attribute 'table'. With it, the compile returns the EXPLAIN's first column as JSON. - A SQL with
'it''s; a \ test', a newline and a literal{{ not jinja }}round-trips exactly:["it's; a \\ test"].
- Without the guard, dbt failed at parse:
- Existing BigQuery tests pass unchanged. The only edited test is the one that listed snowflake and databricks as unsupported; it now uses duckdb, redshift and postgres.
dbt-adapter-accessandtool-clisuites: 150 pass.bun typecheck: green.- Live warehouse: none configured on this machine (no Snowflake or Databricks connection, no Spark runtime). Parsers are tested on the documented formats and were not run against a live warehouse.
Cost side effects
Yes, one warehouse statement per estimate, compile-only, with no data scanned:
- Snowflake:
EXPLAINdoesn't need a running warehouse and uses no compute credits. Per Snowflake's docs it uses cloud-services credits for the compilation, which usually fall within the daily 10% cloud-services adjustment. - Databricks:
EXPLAIN COSTruns on the SQL warehouse or cluster that dbt's profile connects to. If that is stopped, it starts it, and it is billed for the DBUs while it is up (auto-stop applies as usual). - Both: compiling the model renders its Jinja, as before. Macros such as
run_queryin the model can run billed queries, which the tool description already says. One estimate runs per call, and the plugin runs at most one per project at a time.
Screenshots / recordings
N/A (CLI tool output).
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
- I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
🤖 Generated with Claude Code