feat: separate user-facing schema (write/read) from internal model [INFP-234]

Merged
DG
4mo ago
+11645-4166

Why

The user-facing schema was a direct dump of Infrahub's internal model: it advertised internal-only fields (e.g. inherited) as settable and described constrained fields (e.g. attribute kind) as free-form strings. Users — and especially LLM agents — produced technically-invalid schemas as a result.

Goal: generate a distinct write contract (exactly what a user may submit) and read contract (adds visible-but-not-settable fields) from the single source of truth, validate submissions against the write contract at the load boundary, and let clients validate offline via the SDK.

Non-goals (this PR): retiring the SDK's hand-written schema models, publishing the write contract in the REST OpenAPI, and extracting schema code generation off the attribute definition model — all three documented as follow-ups in dev/specs/archive/002-user-facing-schema/opsmill-implement-followups.md.

Closes INFP-234

What changed

Behavioral — POST /api/schema/load now validates every submitted node, generic, and extension against the user-facing write contract:

  • Invalid settable values are rejected with a field-level error naming the dotted path and the offending value: a constrained field outside its allowed set (attribute kind, relationship kind/cardinality, …), a value outside its allowed range or pattern, or a missing required field. Attribute kind and the other bounded fields are now published enums rather than free-form strings, so a kind that does not exist is caught at the boundary instead of failing later.
  • Read-only, internal, and unknown fields are tolerated and dropped, at every nesting level — a schema read back from GET /api/schema, an export, or an older/hand-edited payload still loads without stripping anything. This is a change of direction taken during review: the first implementation rejected those fields, which made the read-back → edit → load round trip a breaking workflow for no benefit. Requirement FR-003 in spec.md still states the original reject rule and has not been rewritten.
  • version is required on the write contract. The server already required it; the generated SDK model did not, so offline validation used to pass payloads the server rejected.
  • Schemas can be validated offline with only the SDK installed, and the offline verdict reproduces the server's.

Implementation

  • New Visibility axis on schema field definitions; every field classified write/read/internal from the single source internal.py.
  • Generator emits write/read model families into the SDK (dedicated generate_schema_sdk.j2), driven by SdkSchemaGenerator in tasks/backend.py; internal models unchanged (byte-identical output).
  • The load boundary is a model_validator(mode="before") on SchemaLoadAPI delegating to the SDK's validate_schema — one implementation, so server and SDK cannot drift. The generated write models are extra="ignore", so pydantic itself drops non-write fields; there is no hand-rolled projection step.
  • Reserved-namespace and reserved-suffix checks unified behind SchemaRoot.validate_reserved_names().
  • Incidental: generator no longer breaks on worktree paths containing shell/regex metacharacters.

Unchanged: GET /api/schema response shape; stored-schema read-back; no database or migration changes.

Suggested review order

  1. backend/infrahub/core/schema/definitions/internal.py — the Visibility axis + per-field classification.
  2. tasks/backend.py + backend/templates/generate_schema_sdk.j2 — write/read generation into the SDK.
  3. backend/infrahub/api/schema.py — the model_validator(mode="before") boundary gate.
  4. Tests, then generated SDK models (mechanical).

How to review

Focus on the boundary gate and the visibility classification. Companion SDK PR (land first): https://github.com/opsmill/infrahub-sdk-python/pull/1135.

How to test

uv run pytest backend/tests/unit/api/test_schema_load_contract.py
uv run pytest backend/tests/unit/core/schema/test_generated_visibility.py
uv run pytest backend/tests/functional/api/test_load_schema.py   # requires Neo4j testcontainers
uv run pytest backend/tests/component/api/test_40_schema.py

Full functional load suite: 17 passed locally (Neo4j testcontainers).

Impact & rollout

  • Backward compatibility: a payload is rejected only for an invalid settable value — an out-of-enum or out-of-range value, or a missing required field. Payloads carrying derived or internal fields (inherited, used_by, hierarchy, derived kind, …) keep loading, so the GET → edit → POST round trip needs no stripping. Changelog fragments and an upgrade note are included.
  • Performance: negligible — schema load is infrequent and not a hot path.
  • Config/env changes: none.
  • Deployment notes: requires a coordinated SDK release; the submodule pointer references the SDK branch, so land the companion SDK PR first.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (upgrade note)
  • Internal .md docs updated (dev/knowledge)
  • I have reviewed AI generated content

🤖 Generated with Claude Code


Summary by cubic

Separates the user-facing schema into SDK-generated write/read models, wires /api/schema and /api/schema/{load,check} to those contracts, and serves the generated read models directly. /api/schema/load drops read‑only/internal/unknown fields, requires version, and validates constrained values with dotted, field‑level errors. Addresses INFP‑234.

  • New Features

    • Generated infrahub_sdk models: InfrahubSchemaWrite/InfrahubSchemaRead; discriminated Attribute union; typed computed_attribute and dropdown choices; shared enums; read models compute kind and include hash; profile/template read models; write‑only extensions.
    • API/OpenAPI: GET /api/schema returns the generated read models (NodeSchemaRead, GenericSchemaRead, ProfileSchemaRead, TemplateSchemaRead); POST /api/schema/{load,check} declares InfrahubSchemaWrite and validates via infrahub_sdk.schema.validate_schema() (extras are dropped). OpenAPI and frontend REST types regenerated; CI guards generated SDK schema/protocol drift.
    • Internals: added Visibility on field definitions to drive write/read generation; removed the hand‑rolled write projection; unified reserved‑name checks under validate_reserved_names(); ADR 0010 recorded; corrected code‑generation docs.
  • Migration

    • You can submit schemas that include read/internal/unknown fields (e.g., a read‑back/export); the server strips them, then enforces the write contract. version is now required.
    • Validate offline with infrahub_sdk.schema.validate_schema(); server and SDK give the same verdicts.
    • The deprecated attribute kind String is removed from SDK read models and OpenAPI; use Text.