feat: add data-vault skill

AL
2mo ago
+19193-0

Introduces .opencode/skills/data-vault covering DV 2.0 hub/link/satellite modeling, hashing discipline, hard/soft rule separation, PIT/bridge, business vault, metrics/error vault, and real-time patterns — with per-platform references for Snowflake, Databricks, BigQuery, Redshift, MS Fabric, and Postgres, plus native-SQL orchestration for non-dbt teams.

Adds an enterprise-delivery layer: governance/compliance (right-to-erasure via crypto-shredding, PII placement, classification propagation), CDC and streaming sources (Debezium/Fivetran/Kafka/paginated APIs), multi-tenancy (tenant-scoped hash keys, RLS, cross-tenant same-as prohibition), data contracts and change management (contract-first ingestion, shared-hub CODEOWNERS + blast-radius review), MDM integration (Informatica/Reltio/Profisee golden-record consumption), and scale/ops (archiving, rehashing playbook, CI cost tiers, DR/backup for insert-only, SLA registry). SKILL.md now runs a Detect → Probe → Ask → Model → Discover → Generate → Load → Validate workflow that explicitly probes enterprise context in Step 0.5 and includes a Never-Assume checklist covering every load-bearing design decision.

Ships alongside experiments/data_vault_skill_validation with six eval suites (skill_structure, sql_correctness, knowledge_qa, platform_routing, agent_tasks, e2e_dbt) totaling 200 assertions run offline in ~30s. e2e_dbt covers 60 end-to-end DV assertions against a real dbt+DuckDB project across 10 scenarios (change-detection, idempotency, multi-source hub, composite key, normalization, hash consistency, ghost row, effectivity satellite, batch atomicity, PIT). CI workflow runs the full suite on any change under .opencode/skills/data-vault/**.

Issue for this PR

Closes #

Type of change

  • New feature

What does this PR do?

Adds a new built-in skill data-vault that gives altimate-code equivalent depth for Data Vault 2.0 that it already has for medallion/Kimball dbt work. Customer feedback said the existing skills produced silently-wrong models on DV 2.0 because the standard staging → mart patterns violate DV's insert-only, hash-key, and audit-metadata contracts.

The skill is trigger-loaded on any dbt_project.yml-containing project when the user's prompt mentions DV terminology (hub, link, satellite, hashdiff, raw vault, PIT, etc.) — same auto-load mechanism as dbt-develop. Its Step 0 asks which warehouse + orchestration tool the customer uses and routes to the correct per-platform reference (Snowflake / Databricks / BigQuery / Redshift / MS Fabric / Postgres); Step 0.5 probes for enterprise context (compliance, multi-tenancy, CDC, shared hubs, MDM, scale) and loads additional references only when relevant.

Every load-bearing design decision (hash algorithm, case convention, NULL sentinel, business-key choice, driver keys, satellite splits, delivery model) is explicitly flagged in a "Never Assume — Ask Before Making These Decisions" section so the skill does not silently default on choices that are retroactively expensive to fix.

The skill's guidance is grounded in Linstedt & Olschimke's Building a Scalable Data Warehouse with Data Vault 2.0 and codifies patterns from AutomateDV / datavault4dbt where those packages already encode correct defaults.

How did you verify your code works?

  • Local eval suite: six independent suites (skill_structure, sql_correctness, knowledge_qa, platform_routing, agent_tasks, e2e_dbt) with 200 assertions total, all passing in ~30s offline. The e2e_dbt suite scaffolds a real dbt project with dbt-duckdb, seeds CSV sources, runs dbt seed + dbt build, executes DV invariant queries against the materialized tables (no duplicate hash keys, PK unique on (parent_hk, load_dts), FK integrity link→hub, hashdiff never NULL, atomic load_dts, ghost-row present, effectivity open/close correctness, PIT row counts), and verifies idempotency via a second dbt run.
  • Real DV bugs caught and fixed by the eval: the initial satellite pattern didn't dedupe source-side, which failed the "PK unique on (parent_hk, load_dts)" assertion when the source contained duplicate rows. Fix committed alongside; regression test now guards it. A second-round bug-hunt caught 4 more (dialect leaks in dialect-neutral column descriptions, missing QUALIFY-not-supported warnings in postgres-specific.md / redshift-specific.md, one mis-tagged code block); all fixed and covered.
  • Binary sanity check: built the skill into a local altimate-code binary via bun run script/build.ts --single; ran altimate-code debug skill from an unrelated directory; confirmed data-vault loads from ~/.altimate/builtin/data-vault/SKILL.md alongside the other 22 skills with correct frontmatter.
  • CI: new workflow .github/workflows/eval-data-vault-skill.yml runs the full suite on any push/PR touching the skill or eval directories; scoped path filter so unrelated PRs don't run it; installs dbt-core + dbt-duckdb so e2e_dbt executes.

Screenshots / recordings

N/A — no UI changes.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Summary by cubic

Adds a built-in data-vault skill that generates correct Data Vault 2.0 models instead of the silently-wrong ones the existing skills produced, and ships a validation suite that catches regressions.

New Features

  • Covers hub/link/satellite modeling, hashing discipline, hard/soft rule separation, PIT/bridge, business vault, and real-time patterns.
  • Routes per-platform references for Snowflake, Databricks, BigQuery, Redshift, MS Fabric, and Postgres, plus native-SQL orchestration for non-dbt teams.
  • Probes warehouse, orchestration, and enterprise context (compliance, multi-tenancy, CDC, MDM, scale) before generating, and flags every load-bearing decision in a Never-Assume checklist.
  • Adds six offline eval suites (skill_structure, sql_correctness, knowledge_qa, platform_routing, agent_tasks, e2e_dbt) with 200 assertions, including a real dbt+DuckDB project across 10 scenarios.
  • Adds a CI workflow that runs the full suite on any change under .opencode/skills/data-vault/**.

Summary by CodeRabbit

  • New Features

    • Added a comprehensive Data Vault 2.0 modeling skill with guidance for hubs, links, satellites, loading, governance, CDC, multi-tenancy, and major data platforms.
    • Added platform-specific and operational reference documentation, including real-time processing, compliance, MDM, PIT tables, and information marts.

  • Tests

    • Added a validation suite covering documentation structure, SQL correctness, knowledge checks, platform guidance, agent tasks, and end-to-end dbt scenarios.

  • Chores

    • Added automated evaluation on relevant changes, with results preserved for review.