feat(schema): collapse entity-per-table warehouses in schema_index

Closed
SA
5mo ago
+1661-20

Closes #760

Motivation

When a database has many tables that share the same column structure — one ticker per table, one tenant per table, time-partitioned tables, per-region tables — schema_index today emits redundant per-table schemas. The agent's context fills with N copies of the same shape, and schema_index's truncation cap means the agent often can't see all tables at all.

This is a real production pattern across:

  • Multi-tenant SaaS (one table per customer for isolation)
  • Time-partitioned analytics (events_2024_01_01, events_2024_01_02, …)
  • Per-region tables (orders_us, orders_eu, orders_apac)
  • Per-environment schemas (dev/staging/prod with same shape)
  • Per-source ingestion (events_segment, events_amplitude, …)

For warehouses that exhibit it, current behaviour wastes context on duplicate schemas and silently hides tail tables beyond the truncation cap.

What's added

A heuristic in schema_index's scan logic that detects the entity-per-table pattern at scan time and emits a composite digest — one shared schema description plus the full list of table names — instead of N redundant schemas. Tables that don't fit the dominant fingerprint still get per-table schemas as before. schema_search is updated to surface entity-group matches.

Detection rule: group tables by sorted (column_name, column_type) fingerprint. If any one fingerprint covers ≥50% of tables AND ≥20 tables, treat it as the entity group and emit a composite digest. Tables with empty column lists are ignored. Both thresholds are overridable per-call via indexWarehouse options.

Composite digest format:

schema: public
pattern: entity-per-table
table_count: 30
composite_columns: [{name: "Close", type: "DOUBLE"}, {name: "Date", type: "VARCHAR"}, ...]
sample_table: TICKER_0
table_names: [TICKER_0, TICKER_1, TICKER_10, TICKER_11, ...]

schema_search queries entity-group rows in addition to per-table rows; a query for a specific table name returns just that row from the group, not the entire member list. The tool-level formatter renders matches under a ## Entity-per-table Groups section.

cacheStatus().total_tables and total_columns were updated to fold collapsed counts in, so existing callers see real warehouse counts regardless of digest format. This also fixed a pre-existing scenario where a 1000-table stress test under-reported counts when groups collapsed.

Tests

19 new unit tests in test/altimate/entity-groups.test.ts:

  • 100 same-shape tables → marked as entity
  • 5 mixed-shape tables → not marked
  • 50 same-shape + 50 different-shape → not marked (under 50% threshold)
  • Empty-column tables ignored for fingerprinting
  • Threshold and min-count overrides honoured per-call

7 new integration tests in test/altimate/schema-cache.test.ts:

  • In-memory SQLite fixture with 30 identical-schema tables produces composite digest
  • Pre-existing 5-mixed-table fixture produces identical results to previous behaviour (backwards compat)
  • 1000-table stress test passes (regression fix)

Results:

  • Affected files (entity-groups.test.ts, schema-cache.test.ts, schema-finops-dbt.test.ts, schema-resolver.test.ts): 139 pass / 0 fail
  • Telemetry suite: 46/0 pass; real-tool-simulation: 40/0 pass — no regressions
  • Full opencode suite when merged with #758 + #759: 7421 pass / 0 fail / 494 skip (~88s)
  • bun run typecheck: clean (0 errors after bun install in the worktree)

Backwards compatibility

  • When no fingerprint group qualifies (the common case), entity_groups is undefined and per-table rows are written exactly as before. Verified by integration test against the existing 5-table mixed fixture.
  • The new entity_groups field on SchemaIndexResult and SchemaSearchResult is optional; older consumers see no breaking change.

Files

  • packages/opencode/src/altimate/native/schema/cache.ts (DDL + scan + search + status)
  • packages/opencode/src/altimate/native/schema/entity-groups.ts (new — pure detector, 174 LOC)
  • packages/opencode/src/altimate/native/types.ts (new optional fields + interfaces)
  • packages/opencode/src/altimate/tools/schema-index.ts (output formatter)
  • packages/opencode/src/altimate/tools/schema-search.ts (output formatter)
  • packages/opencode/test/altimate/entity-groups.test.ts (new — 269 LOC)
  • packages/opencode/test/altimate/schema-cache.test.ts (added integration tests)

Summary by cubic

Collapses “entity-per-table” warehouses in schema_index by emitting a single composite digest per schema, cutting redundant context and keeping search accurate. Adds safe search/output and column reconstruction so downstream tools still see all tables; addresses #760.

  • New Features

    • Detects entity-per-table via column-shape fingerprinting (≥50% and ≥20 tables by default); overrides via schema_index params entityRatioThreshold and entityMinTables.
    • Stores qualifying groups once with shared columns and full table_names; non-group tables remain per-table. At most one group is collapsed per schema.
    • schema_search shows entity-group matches and returns only the queried table(s) from a group. Tool output lists groups and truncates huge table_names (head + tail).
  • Bug Fixes

    • listColumns reconstructs columns for collapsed groups so PII and SQL validation see complete schemas; cacheStatus now includes collapsed groups in total_tables and total_columns.
    • Reduced false positives and fingerprint edge cases with bounded group search_text, table-name matching via post-filter, and safe fingerprinting for :/|/missing types.

Summary by CodeRabbit

  • New Features

    • Schema indexing now detects and collapses groups of tables with identical column structures and surfaces them in index/search results.
    • Search and reporting include entity-group matches, counts, and summarized composite column info.
    • Detection thresholds (ratio and minimum tables) are configurable.
  • Behavior Changes

    • Aggregate table/column counts and column listings now include contributions from collapsed entity groups.
  • Tests

    • New unit and integration tests covering detection, indexing, search, listing, and threshold validation.