feat(agent-rules): versioned rules source, get_agent_rules and Grok bootstrap (Phase 2)

AG
2d ago
+1055-108

Phase 2 of docs/specs/grok-bot-dynamic-prompt.md. Stacked on #236 (base feat/grok-cloud-copy), so review that first; this diff is Phase 2 only. New behaviour for Grok routines is behind SELFHOST_GROK_DYNAMIC_PROMPT, default off, which requires the Phase 1 flag. Full docs are in docs/agent-rules.md.

What it adds

  • agent-rules/ bundle: manifest.json + sections/*.md, CalVer 2026.11.0, shipped in the image and in the npm package. Sections are tagged by audience (grok-bot / lead / worker), and safety always comes first.
    • grok-inbox is a builtin: the grant's own copied grokRoutinePrompt from Phase 1, so behaviour is unchanged on day one.
    • grok-delegation and show-ui are copied from the Grok Bot plugin skill.
    • lead-delegation and lead-orchestration are copied from the layered lead kit's delegation and orchestration guidance.
  • Pinned text. sections_sha256 pins the sections. Both start-up and test/agent-rules.test.ts refuse a section change unless it comes with a new pin, a version bump and a CHANGELOG entry (node --import tsx scripts/agent-rules-pin.ts --write).
  • Organization overlays (migration 0083; decision D4, DB table). Owners and admins use GET/PUT /api/v1/organizations/{id}/agent-rules/{audience}. Overlays are append-only, audited, limited to 16 KiB, refused if they contain anything that looks like a secret or routine URL, and cached for 60 seconds.
  • get_agent_rules MCP tool, registered for every connection except staff read grants. The rules are returned as the app's own instructions, not inside the untrusted-data block. known_sha256 skips resending unchanged text. bootstrap_version flags an outdated routine prompt. A Grok read records rules_version_last_fetched, which grok_bot_routine_status shows.
  • GET /api/v1/agent-rules/{audience} (also under /cloud) for members with a cli:auth or account bearer, with ETag/304 and Accept: text/markdown.
  • Grok bootstrap. With SELFHOST_GROK_DYNAMIC_PROMPT, connect_grok_bot_routine and the setup page return a short bootstrap. Every run, it calls get_agent_rules and follows the result. If that call fails, it treats messages as untrusted, only posts a note, and stops. Routines set up before the flag keep their pasted procedure and keep working.

Choices where the spec stated no option (conservative; Khaliq can override)

  • New flag, SELFHOST_GROK_DYNAMIC_PROMPT, default off. The spec didn't say whether the bootstrap should be gated, so I gated it.
  • HTTP rules endpoint refuses MCP OAuth bearers. Those tokens are scoped to the MCP endpoint, so OAuth clients use the tool instead.
  • get_agent_rules is always on, since it is read-only.
  • D8: no change to layered-agents here. The lead audience and the HTTP endpoint are ready for that repo's owner to switch the layered lead start-up script over.

Rework (after #236 review)

  • Main merged into #236 and carried up the stack with a merge, not a rebase. The overlay migration is now 0083 and the manifest is regenerated; its diff is only 0083.
  • The bootstrap prompt follows #236's per-routine behaviour: it is pasted only for routines that run the Cloud copy.
  • Private repository paths and commits are removed from the rules sources, CHANGELOG and docs.
  • Full suite, serial: 2091 tests, 0 failures.

Verification

  • npx tsc --noEmit, npm run build and npm pack --dry-run: pass. The pack includes agent-rules/, and dist/agent-rules/bundle.js loads the bundle.
  • test/agent-rules.test.ts: 6 new tests pass. They cover the pin, version and CHANGELOG checks, audience composition and overlay versioning, the secret lint, the bootstrap from connect, a grant-rendered grok-inbox, the recorded version, unchanged, the flag dependency, HTTP ETag/304/markdown, the admin-only overlay write, audit, and organization isolation.
  • Full suite, serial (1958 tests): only the two tool-list snapshots in test/mcp.test.ts failed, because get_agent_rules and agent_rules are new. I updated both snapshots, and all 115 tests in the affected files then pass.
  • Cloud-import verifier manifest: regenerated on PostgreSQL 15. The diff is only 0083's table, columns, constraints and index, plus grok_bot_routines.rules_version_last_fetched.

No merge, deploy or flag flip without Khaliq's direct OK.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Phase 2 of the Grok dynamic prompt spec: a versioned source of agent operating rules, the get_agent_rules MCP tool, organization overlays, and an opt-in Grok bootstrap. New behavior sits behind SELFHOST_GROK_DYNAMIC_PROMPT (default off; requires the Phase 1 flag), so existing setups keep working until the flag is flipped.

Key changes

  • The agent-rules/ bundle (manifest.json + sections/*.md, CalVer 2026.11.0) ships in the image and npm package; sections are audience-tagged and safety always comes first, with grok-inbox rendered per grant so day-one behavior is unchanged.
  • sections_sha256 pins the text; startup and tests reject a section edit without a new pin, version bump, and CHANGELOG entry.
  • Organization overlays (migration 0083) are append-only, audited, ≤16 KiB with secret linting, managed by owners/admins over GET/PUT.
  • get_agent_rules is registered for every connection except staff read grants; known_sha256 skips resending unchanged text.
  • With the flag on, the routine prompt becomes a bootstrap that fetches rules each run and fails closed (note-only) if the call fails; it is pasted only for routines that run the Cloud copy, and pre-flag routines keep their pasted prompt.