fix: dbt tools keep the project's manifest, time CTEs without dbt start-up, name graph clauses and bad arguments, key the daemon on the Python env
Merged
HK
14h ago+818+818+012345678901234567890123456789-54-54-01234567890123456789
Issue for this PR
Closes #1780
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
These five bugs turned up while running altimate-code tool run against a DuckDB dbt project from the Claude Code plugin.
- Tool runs no longer overwrite
target/manifest.json. The tools that start the dbt adapter (dbt_cte_profile,dbt_sql_graph,dbt_health,dbt_cost_estimate,dbt_lineage, and dbt-routedsql_execute) ran dbt commands that write to the project'starget/. Each command rewritesmanifest.jsonwith compiled SQL for only the nodes it compiled, so after one tool run a project with 19 compiled models had none. Now, in thecoreandcorecommandmodes,register.tscreates the adapter withtargetPath: <target-path>/.altimate-code. dbt-tools'createthen setsDBT_TARGET_PATHin the environment it passes to dbt's processes, and only there, never inprocess.env. dbt and dbt-integration both honour that variable, so the adapter parses, compiles and reads its own manifest in that folder, and the project's manifest is left as the user'sdbt compilewrote it. Readers of the project's manifest (resolveDbtTargetPath, the stale-manifest fallback) are unchanged.dbt_healthtakescatalog.jsonfrom beside the adapter's manifest, else from the project's target.altimate-dbtcommands run for the user still write to the project'starget/, because they don't pass the option. Thefusionandcloudmodes keep writing to the project's target, because dbt-integration 0.3.19 reads<project>/targetthere whatever dbt is told. dbt_cte_profileno longer counts dbt's start-up in a CTE's time.queryMsused to be the whole call, about 2 s of which is a command-line dbt starting. It is now the warehouse time of the count query. That comes from dbt's own timing of theexecutestep in therun_results.jsonthe command wrote. Each count query starts with a comment unique to the call, and the file is used only when it was written after the call started, by ashowwith one node whose compiled code carries that comment. A concurrent count of the same CTE therefore can't lend its time. If that's not available, it is the call's time less a trivialselect 1through the same adapter, measured once per adapter. Each waiting call stops at its own abort. If neither works it is the call's time, labelled as such.totalMskeeps the whole call, andtiming(dbt/baseline/wall) says which measurequeryMsis. The tool description documents this.dbt_sql_graphnames joins, filters, groups and sorts by their clause. The backend gives these nodes no name, so they were shown under their generated keys (Where_1398…). They are now named by their clause on one line, cut at 80 characters:LEFT JOIN payments ON orders.order_id = payments.order_id,INNER JOIN stg_products AS products ON …,WHERE stg_orders.status <> 'returned'. The joined relation is shortened to its last name part, and an alias that only repeats it is dropped. The rest of the clause is left as written. The key stays the node'sid, and edges use it.- Argument errors name the field. A tool defined with zod and no formatter of its own (for example,
impact_analysis) reportedSchemaError(Expected <Declaration>, …).tool-zod-compatnow re-parses the original input and reports each issue by field, for examplechange_type is required (expected one of "remove"|"rename"|"retype"|"add"|"modify")ormodel: Invalid input: expected string, received number.change_typegets no default, because each value gives a different verdict. - A tool daemon started outside the virtualenv no longer serves a caller inside it.
VIRTUAL_ENVandCONDA_PREFIX, which select the Python dbt runs with, are now part of the daemon's socket key, alongside the other variables a daemon reads once at start. An adapter that didn't start or load now names the Python it used and how to fix that: activate the virtualenv or setpythonPath, then start a new session or runtool daemon stop.
How did you verify your code works?
bun typecheck: passes.- Targeted tests:
dbt-sql-graph,dbt-cte-profile,dbt-adapter-access,tool-cli/*,impact-analysis*,dbt-health,dbt-first-execution,dbt-lineage-*,sql-query-deadline,test/tool, the tool API adversarial suites, and all ofpackages/dbt-tools. 1129 + 208 pass. Two failures are unrelated: the dbt e2e test needsdbt-duckdbin the system dbt, and also fails onorigin/main; oneapply_patchtiming test passed when re-run alone. - New tests, each of which fails with
origin/main's source:- dbt-tools
adapter-target-path.test.ts: a stand-in Python records that every dbt process the adapter starts sees the givenDBT_TARGET_PATH, thatprocess.envstays unchanged, and that dbt sees none without the option. - The adapter is created with
<target>/.altimate-code, followingtarget-pathandDBT_TARGET_PATH. dbt_cte_profile: dbt's timing is used. Stale, foreign or non-showrun_results.jsonfiles are rejected, and so is a file without the call's tag. Two concurrent counts of one CTE each keep their own timing, and a waiter on the shared baseline stops at its own abort.- The
fusionandcloudmodes get no separate target, and an unset mode defaults tocoreand does. Zone-less timestamps are read as UTC. The baseline is measured once per adapter, and a failed baseline is measured again on the next call. - Clause names for the backend's real join/filter/group shapes, an
ASinside a condition, truncation, and a node with no clause. impact_analysiswithoutchange_type, or with a bad value or type.- The daemon socket key changes with
VIRTUAL_ENVandCONDA_PREFIX. - The refusal message names the Python.
dbt_healthreads the project's catalog.
- dbt-tools
- Live recheck with a from-source build on a copy of the DuckDB demo project (dbt 1.12.7, dbt-duckdb):
- After
dbt compile, the project's manifest has 19 nodes withcompiled_code. Before: 0 after onedbt_cte_profilerun. After: still 19 afterdbt_cte_profile×3,dbt_sql_graphanddbt_healththrough the daemon. The adapter's files are intarget/.altimate-code/. customers(200k rows): before,2458 ms. After:6 ms on the warehouse (dbt's own timing of the query), 200000 rows; 2384 ms in all with dbt's start-up.order_items(19M rows):15941 ms on the warehouse … 18390 ms in all.- Graph nodes:
id=Where_1400… name=WHERE stg_orders.status <> 'returned',id=Join_1400… name=LEFT JOIN payments ON orders.order_id = payments.order_id. impact_analysis {"model":"stg_orders"}: before,SchemaError(Expected <Declaration>, …). After:change_type is required (expected one of "remove"|"rename"|"retype"|"add"|"modify").- A project with its virtualenv outside the project folder. The daemon was started without
VIRTUAL_ENV, and the caller then activated it. Before:the dbt adapter didn't load the projecton both calls. After: the first call names/usr/bin/python3and the fix, and the activated call gets its own daemon and answerscustomers: 4 ms on the warehouse ….
- After
Cost side effects
Two small ones, and neither grows with data beyond what the profile already ran:
dbt_cte_profilemay run oneselect 1per dbt adapter, and only when dbt reports no timing of its own.- Each count query now carries a unique comment, so a warehouse result cache (Snowflake's, for example) no longer answers a repeated profile of the same CTE. Each profile runs its count query, as a first profile always did. A cached answer would also have reported a time that isn't the CTE's.
Screenshots / recordings
Not a UI change.
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
- I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
🤖 Generated with Claude Code
Summary by CodeRabbit
-
New Features
- CTE profiling now reports query time, total time, and the timing source, using dbt execution data when available.
- SQL graph nodes without backend-provided names can display readable names based on their clauses.
-
Improvements
- Core dbt tools keep artifacts in a separate project subdirectory; Fusion and Cloud retain their existing artifact location.
- dbt adapter startup guidance includes the detected Python executable and steps to resolve environment issues. Different Python environments are handled separately.
- Catalog discovery checks both the manifest location and the project’s configured target directory.
- Validation errors identify affected fields more clearly.
-
Documentation
- Documentation explains Python environment selection and artifact locations.