Feat: generate llms.txt and markdown pages for AI agents

Merged
AN
4d ago
+296-2

Description of your changes

Makes the documentation consumable by AI agents: generates an llms.txt index and a markdown copy of every docs page at build time.

  • docusaurus-plugin-llms (new dev-time plugin) emits llms.txt, llms-full.txt, and a .md twin next to every page's HTML, so agents can fetch <page-url>.md instead of parsing themed HTML. Version output mirrors the site's routing: the root llms.txt indexes the latest release served at /docs/, while next and older versions get their own indexes under /docs/<version>/llms.txt.
  • scripts/llms-postprocess.js (runs via the postbuild hook, so CI and Netlify-style yarn build both trigger it) restructures the indexes into the llmstxt nested layout: CLI man-pages move to /docs/cli/llms.txt and other versions hang off /docs/versions/llms.txt. This keeps every index in the 77–91K character range — under the ~100K budget agents can ingest in one read — and the script fails the build if an index ever crosses it. It also prepends an index pointer to every generated markdown file.
  • src/theme/Root.js adds a visually hidden, server-rendered note on each page telling agents where the index and markdown variants live.

Measured with afdocs (the scanner behind CLOMonitor's agent readiness section) against a local production build: overall 94/100, with content discoverability at 92 and index coverage at 100%. The one category that still fails — Accept: text/markdown content negotiation — is a hosting limitation, not something the repo can fix.

I have:

  • Read and followed KubeVela's contribution process.
  • Update sidebar.js if adding a new page.
  • Run yarn start to ensure the changes has taken effect.

Special notes for your reviewer

No docs pages or sidebar entries change — all new output is generated into build/ (verified with full yarn build runs rather than yarn start, since the plugin and postbuild hook only run on production builds). Build time grows by roughly 1–2 minutes for the markdown generation across versions. The commit is DCO signed off.


Summary by cubic

Makes the KubeVela docs consumable by AI agents: builds an llms.txt index and a markdown copy of every page at docs build time so agents can fetch https://kubevela.io/llms.txt or any page's .md twin instead of parsing styled HTML. Measured 94/100 on the CLOMonitor agent readiness scan (up from no index at all), with content discoverability 92 and index coverage 100%.

Build changes

  • Adds the docusaurus-plugin-llms dependency and configures it to emit per-version indexes matching the site's routing (/docs/ for the latest release, /docs/<version>/ for next and older versions).
  • Adds scripts/llms-postprocess.js as a postbuild hook that restructures the plugin output into the llmstxt nested layout: CLI man-pages move under <version>/cli/llms.txt, per-version indexes under /docs/versions/llms.txt, and every index stays under the ~100K character agent ingestion budget (the script fails the build if one exceeds it).
  • Chains the postprocessor into the standalone Dockerfile build, which runs docusaurus build directly and would otherwise skip yarn's postbuild hook.
  • Prepends a pointer to the root llms.txt on every generated markdown page and adds a visually hidden note to every HTML page via src/theme/Root.js.

No docs pages or sidebar entries change; all output is generated into build/. Build time grows by roughly 1–2 minutes for markdown generation.