feat: annotate traces with `de.*` attributes via metadata channel + procedural classifier

Closed
SA
3mo ago
+1014-4

Summary

Wire the existing trace-attribute scaffolding (de-attributes.ts, setSpanAttributes, viewer grouped rendering at viewer.ts:514) so real values flow onto every tool span and the session root.

Before this change, 0/101 traces on disk had any populated attributes field — the schema slot, vocabulary, setter, and viewer rendering all existed but had zero callers. This PR connects the seam.

Two layers, no LLM:

  • Layer 1 — tool-provided metadata (high fidelity, from real driver/parser values). Tools surface structured fields by setting de.* keys on their returned metadata object. Trace.logToolCall lifts those keys onto span.attributes. Tools never import the observability layer — the seam is the existing ToolStateCompleted.metadata field already plumbed through message-v2. Per-value 10 KB + total 32 KB byte caps prevent runaway payloads.

  • Layer 2 — procedural classifier (observability/annotator.ts). Pure function over (toolName, input, output). Lookup-table tool taxonomy, regex bash-intent + dbt-layer-from-path classification, deterministic outcome/artifacts/env rollups. Best-effort — returns undefined rather than emitting wrong attributes.

Layer 1 wins over Layer 2 on key conflicts in both per-tool and session merges.

What got added (didn't exist before)

1. Tracer reads tool metadata (tracing.ts:logToolCall)

state.metadata is filtered to keys starting with de., JSON-validated per value, and merged into span.attributes. Bounded by per-value (10 KB) and total (32 KB) byte caps so a giant de.sql.query_text or lineage array can't balloon snapshots.

2. Procedural classifier (observability/annotator.ts, new file)

Two pure functions:

  • annotateToolSpan(name, input, output) — derives de.tool.*, de.dbt.layer, de.dbt.command, de.sql.query_text, de.sql.lineage.input_tables.
  • annotateSession(trace) — derives de.workflow.type (with confidence), de.outcome.*, de.artifacts.*, de.env.*.

Called from logToolCall (per-span), endTrace (session rollup), and flushSync (crash-path session rollup). Best-effort, never throws.

3. Five new sub-namespaces in de-attributes.ts

NamespaceKeys
DE.WORKFLOWtype, intent, type_confidence
DE.OUTCOMEclass, executed, change_applied
DE.ARTIFACTSfiles_read, files_edited, models_mentioned, tables_referenced
DE.ENVdbt_present, dbt_manifest_present, warehouse_type, tools_detected
DE.TOOLcategory, subcategory, vendor, bash_intent, bash_invoked

Plus two new keys in existing namespaces:

  • de.dbt.layer (staging / intermediate / dim / fact / agg / mart / source / seed / macro / test / snapshot)
  • de.warehouse.rows_total (for schema_inspect's table-level row count — distinct from query-result rows_returned)

4. Viewer grouping extension (viewer.ts:514)

Added 5 color-coded sections (Workflow, Outcome, Artifacts, Environment, Tool) so the new attributes render with their own labeled groups in the side panel instead of falling through to the generic key-value bucket.

5. Four tool opt-ins (Layer 1, high-fidelity)

ToolEmits
sql_executede.warehouse.{system,rows_returned}, de.sql.dialect (resolved from Registry warehouse)
altimate_core_column_lineageStructured de.sql.lineage.{input_tables,output_table,columns_read,columns_written} from altimate-core's parser — preserves case via structured endpoint extraction (no dot-split that would corrupt quoted identifiers)
schema_inspectde.warehouse.rows_total, de.sql.lineage.output_table
project_scande.env.{dbt_present,dbt_manifest_present,warehouse_type,tools_detected} from authoritative scan results

These four tools never import the observability layer — they set de.* keys on their existing returned metadata object.

What this PR explicitly does NOT touch

  • altimate-core / altimate-core-internal — unmodified.
  • Tool framework (tool/tool.ts) — the metadata channel was already there.
  • Message schema (message-v2.ts) — already carried state.metadata.
  • TraceFile / TraceSpan schema — already had the attributes slot.
  • The other ~70 tools — covered by Layer 2 (procedural classifier).

Codex review

Each chunk was reviewed by Codex before moving on. Fixes folded into this PR:

  • Relative dbt-layer path matching (normalize backslashes, anchor at (?:^|/)models/...)
  • SQL-text capping uses slice instead of dropping the value
  • Workflow classifier counts dbt-bash spans by de.tool.bash_intent attribute, not all bash spans
  • Scalar output_table (omit on multi-output rather than switch type to array)
  • Structured endpoint extraction for column lineage (uses source_table / source_column when present)
  • flushSync runs the session rollup so crash/interrupted traces get root attrs
  • Layered merge direction: explicit setSpanAttributes(..., \"session\") callers still win over derived
  • de.warehouse.rows_total instead of de.quality.row_count for schema-inspect totals (quality keys reserved for profiling/check results)

Test plan

  • bun run typecheck — clean (pre-existing zod/effect/datamate noise unrelated)
  • Local build via bun run build:local succeeds; binary runs against the Altimate gateway
  • E2E live run: prompt that exercises read (dbt layer), bash (dbt build + wc -c), and a final assistant turn. Trace shows:
    • read span: de.tool.category=fs, de.dbt.layer=staging
    • bash span: de.tool.bash_intent=dbt, de.dbt.command=build
    • Session root: de.outcome.class=success, de.outcome.executed=true, de.artifacts.files_read=[...]
  • Retroactive validation over the 101 historical traces via a one-off pure-function pass:
    • 103/104 files annotated, 0 failed
    • 3010/3016 tool spans (99.8 %) got new de.* attributes
    • 102 sessions got root-level rollup
    • Workflow distribution: 51 dbt_develop, 22 dbt_troubleshoot, 8 warehouse_exploration
    • Outcome distribution: 89 success, 11 failure, 2 interrupted (matches the original summary.status mix)
    • Real DE tool spans (sql_execute, dbt_profiles, warehouse_list, warehouse_add, schema_inspect) all received their de.tool.* and de.sql.* attributes via the procedural classifier
  • CI typecheck + bridge-invariant tests
  • Review the changes against bridge-guard if any upstream sync is pending

Trace size impact

Per-tool-span attributes: bounded by the 32 KB total cap implemented in logToolCall.
Session root attributes: small fixed shape (workflow code, outcome enum, file path arrays capped at 100 entries each).
Existing MAX_SERIALIZED_SPANS=5000 cap is unchanged.


Summary by cubic

Annotates every tool span and the session root with structured de.* attributes via a tool metadata channel and a deterministic annotator. Adds new attribute namespaces and viewer groups for workflow, outcome, artifacts, environment, and tool taxonomy.

  • New Features

    • Lift de.* keys from tool metadata onto span attributes with 10 KB per-value and 32 KB total UTF-8 byte caps; preserves caller-set attributes.
    • Add procedural annotator to infer tool taxonomy, bash intent (command-boundary anchored), dbt layer, SQL query/input tables across SQL tools, and session workflow/outcome/artifacts/env; tool-provided values win on conflicts.
    • Extend de-attributes.ts with WORKFLOW, OUTCOME, ARTIFACTS, ENV, TOOL, plus de.dbt.layer and de.warehouse.rows_total; update viewer groups.
    • Tool opt-ins: sql_execute (de.warehouse.{system,rows_returned}, de.sql.dialect), altimate_core_column_lineage (structured lineage with object-form endpoints), schema_inspect (de.warehouse.rows_total, de.sql.lineage.output_table), project_scan (de.env.*).
  • Bug Fixes

    • Bash intent: classify altimate-dbt before dbt, anchor patterns to command boundaries, and hoist regexes to module scope.
    • SQL extraction: extend to all sql_* and altimate_core_* SQL tools (plus sql_diff tables); remove annotator-side char caps and rely on tracer byte caps; extract bash SQL lineage from the full query.
    • Lineage: guard column_lineage with Array.isArray + record checks; support object-form source_column/target_column; strip table prefix in column fallbacks; keep de.sql.lineage.output_table scalar.
    • Session rollup: runs on end and crash; de.outcome.executed reads cached de.tool.bash_intent/de.dbt.command; de.outcome.change_applied requires a successful write/edit; workflow counts only bash spans with de.tool.bash_intent.
    • Environment: derive de.env.dbt_manifest_present from file existence; read de.env.* from the latest project_scan span (prefer Layer 1 over text); switch tool opt-ins to DE_* constants.

Summary by CodeRabbit

  • New Features
    • Traces now automatically enrich tool spans and root sessions with best-effort metadata, including tool taxonomy, bash/dbt intent, dbt layer detection, and SQL input table references.
    • Added new Data Engineering trace attributes for workflow/outcome, artifacts, environment capabilities, and tool classification, and expanded DE grouping in the trace viewer.
    • Enhanced tools to emit richer observability metadata: column lineage, environment capability signals, row totals/lineage output tables, and resolved warehouse system/dialect for SQL runs.
  • Bug Fixes
    • Improved crash handling to prevent corrupted/overwritten crashed trace exports during overlapping runs.