feat(marketing,api,docs): agent-readiness surfaces — markdown twins, MCP discovery, agent auth metadata

Merged
SA
4mo ago
+1178-79

Links

Summary

  • Make superset.sh discoverable and usable by AI agents: markdown surfaces for every content page, MCP server discovery via .well-known, an agent auth walkthrough wired into the OAuth metadata, and structured JSON API errors.
  • Everything machine-readable is either generated from existing constants/content or introspected from the live MCP server — no hand-maintained catalogs.

Why / Context

An orank agent-readiness scan graded superset.sh 59/100: agents landing on the site couldn't get markdown, couldn't discover the MCP server from well-known paths, couldn't learn the auth flow without reading human docs, and got HTML error pages from the API. This PR closes the codebase-fixable gaps; external items (Wikipedia entity, ChatGPT directory, skills.sh publishing) are listed in the plan doc.

How It Works

Marketing (superset.sh)

  • src/lib/llms.ts is the single builder behind llms.txt/llms.md — adds "When to use Superset" and a developer-resource index (MCP server, auth.md, CLI, SDK, scoped llms.txt files).
  • New proxy.ts (Next 16) rewrites .md-suffixed blog/compare URLs to a markdown route, serves Accept: text/markdown negotiation on / and content pages, and /?mode=agent. Markdown twins derive from the same MDX sources (/md/[...path] strips frontmatter/JSX).
  • Static agent surfaces: /index.md, /agents.md, /auth.md (WorkOS auth.md convention: Discover → Register → Claim → Use → Errors → Revocation, all real endpoints).
  • robots.txt route (replaces robots.ts): explicit allows for AI search/user agents, blocks CCBot/Bytespider, Content-Signal line.
  • .well-known/: A2A agent card (honest — states we speak MCP, not A2A), MCP discovery + server card, RFC 9727 api-catalog. RFC 8288 Link headers on /.
  • JSON-LD: Organization gains contactPoint/address/sameAs; homepage gains WebPage speakable + Service. Contact page gains direct-channel prose.

API (api.superset.sh)

  • /.well-known/mcp + /.well-known/mcp/server-card.json: card introspects the running mcp-v2 server via an in-memory MCP client (tools/list), so the 27-tool catalog can never drift from code. Marketing proxies this card with a static fallback.
  • AS metadata (RFC 8414) gains an agent_auth block (skill → auth.md, register/revocation URIs read from better-auth's own metadata). PRM gains resource_documentation.
  • /api root returns 401 JSON with spec-shaped WWW-Authenticate: Bearer resource_metadata=...; unmatched /api/* paths return JSON 404s with hints instead of the Next HTML 404.

Docs (docs.superset.sh)

  • /llms.txt page index (docs already had /llms-full.txt and per-page /llms.mdx/*).

Manual QA Checklist

Validated locally against dev servers of all three apps, plus production builds (next build && next start) of marketing and api:

  • llms.txt, llms.md, index.md, agents.md, auth.md, robots.txt serve 200 with correct content-types (text/markdown where applicable)
  • Accept: text/markdown on / and blog/compare pages returns markdown with Vary: Accept; plain requests still get HTML; /blog/llms.txt not hijacked by negotiation
  • /blog/<slug>.md and /compare/<slug>.md twins return frontmatter-stripped markdown; bad slugs 404
  • /?mode=agent returns the agent view
  • .well-known routes: agent-card, mcp, server-card (27 tools introspected), api-catalog with application/linkset+json;profile=... content-type
  • AS metadata carries agent_auth; PRM carries resource_documentation; existing fields unchanged
  • /api → 401 JSON + WWW-Authenticate; /api/v1 → JSON 404; real routes not shadowed (MCP endpoint still 401s correctly, better-auth register still responds)
  • Homepage renders all JSON-LD types (Organization/ContactPoint/PostalAddress/Service/WebPage/Speakable/FAQPage)
  • docs /llms.txt lists all pages

Testing

  • bun run typecheck (marketing, api, docs)
  • bun run lint (exits 0)
  • bun run build (marketing + api production builds pass)

Design Decisions

  • Live introspection over a static tool catalog: the server card builds from tools/list on the running server; zero drift, no duplicated descriptions.
  • Dropped a hand-authored /openapi.json (shipped mid-branch, then reverted): our API is JSON-RPC over one endpoint, so an OpenAPI spec reduces to a single generic operation no function-calling framework can use. service-desc links point at the MCP server card instead. Forfeits ~7-9 scanner points deliberately.
  • Also dropped: NLWeb schemamap (unverifiable format), 401 stubs at nonexistent paths (/v1, /v2), Web Bot Auth key directory (keys nothing signs with). Details in the plan doc.
  • robots ai-train=yes: being in training data helps LLMs recommend Superset; CCBot/Bytespider blocked as bulk scrapers.

Known Limitations

  • Homepage HTML variant can't carry Vary: Accept — Next.js owns the Vary header for HTML routes and drops middleware/config overrides (verified in next start). Markdown responses do carry it; / renders dynamically so cache-poisoning risk is low.
  • Marketing's server-card proxies the api card at request time (1h revalidate) with a static fallback until the api deploy is live.
  • agent_auth intentionally omits the draft-spec identity_types_supported ceremony we don't implement.

Risks / Rollout

  • Risk: new marketing proxy.ts runs on /, /blog/:slug, /compare/:slug. Matcher is narrow and pass-through by default; segments with extensions are excluded from negotiation.
  • Risk: /api/* JSON 404 catch-all answers previously-HTML 404s; defined routes always win, verified live routes unaffected.
  • Rollback: revert the PR — all changes are additive routes/headers; no data or schema changes.

Summary by cubic

Makes superset.sh discoverable and usable by agents. Adds markdown twins and content negotiation, .well-known MCP discovery with a live server card, an auth.md walkthrough wired into OAuth metadata, and JSON API errors.

  • New Features
    • Marketing: machine-readable markdown pages (/index.md, /agents.md, /auth.md), .md twins and Accept: text/markdown negotiation (/?mode=agent); discovery at /.well-known (MCP registry + live server card with static fallback, agent-card.json, RFC 9727 api-catalog, Link headers on /); rebuilt llms.txt and new llms.md; updated robots.txt to welcome AI crawlers; richer Organization/WebPage/Service JSON‑LD and contact info.
    • API: /.well-known/mcp and a server card that introspects the running @superset/mcp-v2 server; OAuth metadata adds agent_auth and PRM adds resource_documentation; /api now returns spec-shaped 401 with WWW-Authenticate and JSON 404 with hints.
    • Docs: new /llms.txt index of pages.

Summary by CodeRabbit

  • New Features
    • Added machine-readable MCP and agent discovery endpoints, including server cards, API catalogs, and authentication metadata.
    • Added Markdown and plain-text resources for AI-readable documentation, blog posts, comparisons, and site content.
    • Added OAuth guidance, agent capabilities documentation, and clearer authentication responses.
    • Added crawler directives, structured data, and improved contact information on the marketing site.
  • Improvements
    • Added content negotiation for Markdown responses and resilient MCP server-card delivery.
    • Added consistent public caching and cross-origin access for discovery resources.