feat(schema): collapse entity-per-table warehouses in schema_index
Closes #760
Motivation
When a database has many tables that share the same column structure — one ticker per table, one tenant per table, time-partitioned tables, per-region tables — schema_index today emits redundant per-table schemas. The agent's context fills with N copies of the same shape, and schema_index's truncation cap means the agent often can't see all tables at all.
This is a real production pattern across:
- Multi-tenant SaaS (one table per customer for isolation)
- Time-partitioned analytics (
events_2024_01_01,events_2024_01_02, …) - Per-region tables (
orders_us,orders_eu,orders_apac) - Per-environment schemas (dev/staging/prod with same shape)
- Per-source ingestion (
events_segment,events_amplitude, …)
For warehouses that exhibit it, current behaviour wastes context on duplicate schemas and silently hides tail tables beyond the truncation cap.
What's added
A heuristic in schema_index's scan logic that detects the entity-per-table pattern at scan time and emits a composite digest — one shared schema description plus the full list of table names — instead of N redundant schemas. Tables that don't fit the dominant fingerprint still get per-table schemas as before. schema_search is updated to surface entity-group matches.
Detection rule: group tables by sorted (column_name, column_type) fingerprint. If any one fingerprint covers ≥50% of tables AND ≥20 tables, treat it as the entity group and emit a composite digest. Tables with empty column lists are ignored. Both thresholds are overridable per-call via indexWarehouse options.
Composite digest format:
schema: public
pattern: entity-per-table
table_count: 30
composite_columns: [{name: "Close", type: "DOUBLE"}, {name: "Date", type: "VARCHAR"}, ...]
sample_table: TICKER_0
table_names: [TICKER_0, TICKER_1, TICKER_10, TICKER_11, ...]
schema_search queries entity-group rows in addition to per-table rows; a query for a specific table name returns just that row from the group, not the entire member list. The tool-level formatter renders matches under a ## Entity-per-table Groups section.
cacheStatus().total_tables and total_columns were updated to fold collapsed counts in, so existing callers see real warehouse counts regardless of digest format. This also fixed a pre-existing scenario where a 1000-table stress test under-reported counts when groups collapsed.
Tests
19 new unit tests in test/altimate/entity-groups.test.ts:
- 100 same-shape tables → marked as entity
- 5 mixed-shape tables → not marked
- 50 same-shape + 50 different-shape → not marked (under 50% threshold)
- Empty-column tables ignored for fingerprinting
- Threshold and min-count overrides honoured per-call
7 new integration tests in test/altimate/schema-cache.test.ts:
- In-memory SQLite fixture with 30 identical-schema tables produces composite digest
- Pre-existing 5-mixed-table fixture produces identical results to previous behaviour (backwards compat)
- 1000-table stress test passes (regression fix)
Results:
- Affected files (
entity-groups.test.ts,schema-cache.test.ts,schema-finops-dbt.test.ts,schema-resolver.test.ts): 139 pass / 0 fail - Telemetry suite: 46/0 pass;
real-tool-simulation: 40/0 pass — no regressions - Full opencode suite when merged with #758 + #759: 7421 pass / 0 fail / 494 skip (~88s)
bun run typecheck: clean (0 errors afterbun installin the worktree)
Backwards compatibility
- When no fingerprint group qualifies (the common case),
entity_groupsisundefinedand per-table rows are written exactly as before. Verified by integration test against the existing 5-table mixed fixture. - The new
entity_groupsfield onSchemaIndexResultandSchemaSearchResultis optional; older consumers see no breaking change.
Files
packages/opencode/src/altimate/native/schema/cache.ts(DDL + scan + search + status)packages/opencode/src/altimate/native/schema/entity-groups.ts(new — pure detector, 174 LOC)packages/opencode/src/altimate/native/types.ts(new optional fields + interfaces)packages/opencode/src/altimate/tools/schema-index.ts(output formatter)packages/opencode/src/altimate/tools/schema-search.ts(output formatter)packages/opencode/test/altimate/entity-groups.test.ts(new — 269 LOC)packages/opencode/test/altimate/schema-cache.test.ts(added integration tests)
Summary by cubic
Collapses “entity-per-table” warehouses in schema_index by emitting a single composite digest per schema, cutting redundant context and keeping search accurate. Adds safe search/output and column reconstruction so downstream tools still see all tables; addresses #760.
-
New Features
- Detects entity-per-table via column-shape fingerprinting (≥50% and ≥20 tables by default); overrides via
schema_indexparamsentityRatioThresholdandentityMinTables. - Stores qualifying groups once with shared columns and full
table_names; non-group tables remain per-table. At most one group is collapsed per schema. schema_searchshows entity-group matches and returns only the queried table(s) from a group. Tool output lists groups and truncates hugetable_names(head + tail).
- Detects entity-per-table via column-shape fingerprinting (≥50% and ≥20 tables by default); overrides via
-
Bug Fixes
listColumnsreconstructs columns for collapsed groups so PII and SQL validation see complete schemas;cacheStatusnow includes collapsed groups intotal_tablesandtotal_columns.- Reduced false positives and fingerprint edge cases with bounded group
search_text, table-name matching via post-filter, and safe fingerprinting for:/|/missing types.
Summary by CodeRabbit
-
New Features
- Schema indexing now detects and collapses groups of tables with identical column structures and surfaces them in index/search results.
- Search and reporting include entity-group matches, counts, and summarized composite column info.
- Detection thresholds (ratio and minimum tables) are configurable.
-
Behavior Changes
- Aggregate table/column counts and column listings now include contributions from collapsed entity groups.
-
Tests
- New unit and integration tests covering detection, indexing, search, listing, and threshold validation.