fix: dbt tools keep the project's manifest, time CTEs without dbt start-up, name graph clauses and bad arguments, key the daemon on the Python env

Merged
HK
14h ago
+818-54

Issue for this PR

Closes #1780

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

These five bugs turned up while running altimate-code tool run against a DuckDB dbt project from the Claude Code plugin.

  • Tool runs no longer overwrite target/manifest.json. The tools that start the dbt adapter (dbt_cte_profile, dbt_sql_graph, dbt_health, dbt_cost_estimate, dbt_lineage, and dbt-routed sql_execute) ran dbt commands that write to the project's target/. Each command rewrites manifest.json with compiled SQL for only the nodes it compiled, so after one tool run a project with 19 compiled models had none. Now, in the core and corecommand modes, register.ts creates the adapter with targetPath: <target-path>/.altimate-code. dbt-tools' create then sets DBT_TARGET_PATH in the environment it passes to dbt's processes, and only there, never in process.env. dbt and dbt-integration both honour that variable, so the adapter parses, compiles and reads its own manifest in that folder, and the project's manifest is left as the user's dbt compile wrote it. Readers of the project's manifest (resolveDbtTargetPath, the stale-manifest fallback) are unchanged. dbt_health takes catalog.json from beside the adapter's manifest, else from the project's target. altimate-dbt commands run for the user still write to the project's target/, because they don't pass the option. The fusion and cloud modes keep writing to the project's target, because dbt-integration 0.3.19 reads <project>/target there whatever dbt is told.
  • dbt_cte_profile no longer counts dbt's start-up in a CTE's time. queryMs used to be the whole call, about 2 s of which is a command-line dbt starting. It is now the warehouse time of the count query. That comes from dbt's own timing of the execute step in the run_results.json the command wrote. Each count query starts with a comment unique to the call, and the file is used only when it was written after the call started, by a show with one node whose compiled code carries that comment. A concurrent count of the same CTE therefore can't lend its time. If that's not available, it is the call's time less a trivial select 1 through the same adapter, measured once per adapter. Each waiting call stops at its own abort. If neither works it is the call's time, labelled as such. totalMs keeps the whole call, and timing (dbt / baseline / wall) says which measure queryMs is. The tool description documents this.
  • dbt_sql_graph names joins, filters, groups and sorts by their clause. The backend gives these nodes no name, so they were shown under their generated keys (Where_1398…). They are now named by their clause on one line, cut at 80 characters: LEFT JOIN payments ON orders.order_id = payments.order_id, INNER JOIN stg_products AS products ON …, WHERE stg_orders.status <> 'returned'. The joined relation is shortened to its last name part, and an alias that only repeats it is dropped. The rest of the clause is left as written. The key stays the node's id, and edges use it.
  • Argument errors name the field. A tool defined with zod and no formatter of its own (for example, impact_analysis) reported SchemaError(Expected <Declaration>, …). tool-zod-compat now re-parses the original input and reports each issue by field, for example change_type is required (expected one of "remove"|"rename"|"retype"|"add"|"modify") or model: Invalid input: expected string, received number. change_type gets no default, because each value gives a different verdict.
  • A tool daemon started outside the virtualenv no longer serves a caller inside it. VIRTUAL_ENV and CONDA_PREFIX, which select the Python dbt runs with, are now part of the daemon's socket key, alongside the other variables a daemon reads once at start. An adapter that didn't start or load now names the Python it used and how to fix that: activate the virtualenv or set pythonPath, then start a new session or run tool daemon stop.

How did you verify your code works?

  • bun typecheck: passes.
  • Targeted tests: dbt-sql-graph, dbt-cte-profile, dbt-adapter-access, tool-cli/*, impact-analysis*, dbt-health, dbt-first-execution, dbt-lineage-*, sql-query-deadline, test/tool, the tool API adversarial suites, and all of packages/dbt-tools. 1129 + 208 pass. Two failures are unrelated: the dbt e2e test needs dbt-duckdb in the system dbt, and also fails on origin/main; one apply_patch timing test passed when re-run alone.
  • New tests, each of which fails with origin/main's source:
    • dbt-tools adapter-target-path.test.ts: a stand-in Python records that every dbt process the adapter starts sees the given DBT_TARGET_PATH, that process.env stays unchanged, and that dbt sees none without the option.
    • The adapter is created with <target>/.altimate-code, following target-path and DBT_TARGET_PATH.
    • dbt_cte_profile: dbt's timing is used. Stale, foreign or non-show run_results.json files are rejected, and so is a file without the call's tag. Two concurrent counts of one CTE each keep their own timing, and a waiter on the shared baseline stops at its own abort.
    • The fusion and cloud modes get no separate target, and an unset mode defaults to core and does. Zone-less timestamps are read as UTC. The baseline is measured once per adapter, and a failed baseline is measured again on the next call.
    • Clause names for the backend's real join/filter/group shapes, an AS inside a condition, truncation, and a node with no clause.
    • impact_analysis without change_type, or with a bad value or type.
    • The daemon socket key changes with VIRTUAL_ENV and CONDA_PREFIX.
    • The refusal message names the Python.
    • dbt_health reads the project's catalog.
  • Live recheck with a from-source build on a copy of the DuckDB demo project (dbt 1.12.7, dbt-duckdb):
    • After dbt compile, the project's manifest has 19 nodes with compiled_code. Before: 0 after one dbt_cte_profile run. After: still 19 after dbt_cte_profile ×3, dbt_sql_graph and dbt_health through the daemon. The adapter's files are in target/.altimate-code/.
    • customers (200k rows): before, 2458 ms. After: 6 ms on the warehouse (dbt's own timing of the query), 200000 rows; 2384 ms in all with dbt's start-up. order_items (19M rows): 15941 ms on the warehouse … 18390 ms in all.
    • Graph nodes: id=Where_1400… name=WHERE stg_orders.status <> 'returned', id=Join_1400… name=LEFT JOIN payments ON orders.order_id = payments.order_id.
    • impact_analysis {"model":"stg_orders"}: before, SchemaError(Expected <Declaration>, …). After: change_type is required (expected one of "remove"|"rename"|"retype"|"add"|"modify").
    • A project with its virtualenv outside the project folder. The daemon was started without VIRTUAL_ENV, and the caller then activated it. Before: the dbt adapter didn't load the project on both calls. After: the first call names /usr/bin/python3 and the fix, and the activated call gets its own daemon and answers customers: 4 ms on the warehouse ….

Cost side effects

Two small ones, and neither grows with data beyond what the profile already ran:

  • dbt_cte_profile may run one select 1 per dbt adapter, and only when dbt reports no timing of its own.
  • Each count query now carries a unique comment, so a warehouse result cache (Snowflake's, for example) no longer answers a repeated profile of the same CTE. Each profile runs its count query, as a first profile always did. A cached answer would also have reported a time that isn't the CTE's.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • CTE profiling now reports query time, total time, and the timing source, using dbt execution data when available.
    • SQL graph nodes without backend-provided names can display readable names based on their clauses.
  • Improvements

    • Core dbt tools keep artifacts in a separate project subdirectory; Fusion and Cloud retain their existing artifact location.
    • dbt adapter startup guidance includes the detected Python executable and steps to resolve environment issues. Different Python environments are handled separately.
    • Catalog discovery checks both the manifest location and the project’s configured target directory.
    • Validation errors identify affected fields more clearly.
  • Documentation

    • Documentation explains Python environment selection and artifact locations.