feat(sql): cross-DB join key inference via prefix/suffix overlap

Closed
SA
5mo ago
+1293-5

Closes #758

Motivation

When altimate-code is connected to multiple warehouses, columns that point at the same entity often have mismatched naming or formatting — e.g., one DB stores cust_42 while another stores 42, or one warehouse uses businessid_X and another uses businessref_X. There's currently no built-in way to surface candidate joins across DBs; the agent has to discover them per-pair, which costs tokens and is brittle.

Common situations where this lands:

  • Post-acquisition system integration (acquired company's ID format ≠ parent's)
  • Multi-vendor analytics (Salesforce AccountId ↔ Hubspot company_id ↔ internal account_external_id)
  • Migrations leaving prefix differences (v1_user_123 vs user_123)
  • Per-source ingestion with system-specific ID schemes

What's added

A new tool altimate_core.detect_join_candidates that:

  1. Pulls ~50 sample values per string-typed column from each configured warehouse via the existing connector.execute plumbing.
  2. Computes the longest common prefix per column, walking back to the last _/-/: separator (so partial-name matches don't count).
  3. Compares prefix-stripped suffixes pairwise across DBs; emits a candidate when both columns have non-empty distinct prefixes AND share at least one suffix.
  4. Returns ranked candidates: {left_db, left_table, left_col, right_db, right_table, right_col, prefix_rule, suffix_overlap, confidence} where confidence = overlap / min(|left_suffixes|, |right_suffixes|) (cheap and monotonic, deliberately not a probability).

Errors are scoped: per-table sample failures don't kill the scan; per-connection failures are returned in connection_errors. Identifier quoting is ANSI-safe (double quotes). The STRING_TYPE_PATTERN filter restricts sampling to text-like columns (varchar/text/char/string/uuid/json) — numeric/temporal columns are skipped.

Tests

21 new tests in test/altimate/tools/altimate-core-detect-join-candidates.test.ts:

  • commonPrefix LCP logic + separator handling (_/-/:) + no-separator rejection + non-string defensive paths
  • detectJoinCandidatesFromBags: canonical pattern, same-DB rejection, identical-prefix rejection, zero-overlap rejection, no-separator rejection, ranking by overlap/confidence, full N*(N-1)/2 fan-out
  • Integration: two real bun:sqlite :memory: DBs holding 10 businessid_X rows + 8 businessref_X rows (with intentional non-overlapping outliers), driven through the actual native handler with Registry.get stubbed
  • Tool surface: title/output formatting, empty-list rendering, connection-error rendering, dispatcher-throws envelope

Results:

  • This file: 21 pass / 0 fail
  • Sibling tests under test/altimate/tools/: 192 pass / 0 fail (the 9 pre-existing failures in sql-classify.test.ts were verified unrelated by stashing this branch's changes — same 9 fail on main)
  • Full opencode suite when merged with #759 + #760: 7421 pass / 0 fail / 494 skip (~88s)
  • bun run typecheck: clean (0 errors after bun install in the worktree)

The existing altimate-core-native.test.ts count assertion was updated (34 → 35) to reflect the new method, with altimate_core.detect_join_candidates added to the canonical ALL_METHODS list.

Backwards compatibility

Pure addition. No existing behaviour changes; no existing tests modified beyond the count update.

Files

  • packages/opencode/src/altimate/native/connections/detect-join-candidates.ts (new — 317 LOC)
  • packages/opencode/src/altimate/native/connections/register.ts (handler registration)
  • packages/opencode/src/altimate/native/types.ts (params + result types)
  • packages/opencode/src/altimate/tools/altimate-core-detect-join-candidates.ts (new tool wrapper)
  • packages/opencode/src/altimate/index.ts (barrel export)
  • packages/opencode/src/tool/registry.ts (tool registration)
  • packages/opencode/test/altimate/tools/altimate-core-detect-join-candidates.test.ts (new — 385 LOC)
  • packages/opencode/test/altimate/altimate-core-native.test.ts (count assertion update)

Summary by cubic

Adds cross-DB join key inference that finds candidate joins by stripping distinct prefixes and matching suffixes across warehouses. Now uses dialect-aware sampling SQL, stricter permission/error handling, and safer input limits.

  • New Features

    • Exposes altimate_core.detect_join_candidates and the altimate_core_detect_join_candidates tool; registered and exported.
    • Builds sampling SQL with dialect-aware quoting via quoteIdentForDialect; no inline LIMITs (caps passed to drivers).
    • Prefix logic trims to _/-/:; prefixes must differ; emits on suffix overlap; ranks by suffix_overlap, then match_score = overlap / min(left, right).
    • Samples ~50 string-like values/column; scans up to 50 tables/connection; optional schema_name; cross-DB only; skips non-string columns.
    • Robustness: gated by sql_execute_read permission; parallel per-connection scans; returns connection_errors and bounded partial_errors; input caps (connections ≤ 16, sample_size ≤ 1000, max_tables_per_connection ≤ 500).
  • Bug Fixes

    • Registration tests now load connections/register and assert against the advertised method list instead of a hard-coded count, ensuring altimate_core.detect_join_candidates is reliably registered.

Summary by CodeRabbit

  • New Features
    • Added an Altimate tool that detects and ranks cross-database join candidates, with sampling, dialect-aware SQL handling, and clear success/failure results.
  • Chores
    • Made the detection capability available through the native bridge and tool registry for discovery and use; improved shared dialect quoting utility.
  • Tests
    • Added extensive unit and integration tests covering detection logic, sampling, dialect quoting, permissions, and tool execution paths.