Fix site indexability with per-route prerendering

Merged
KA
2mo ago
+475-70

Every URL was served one index.html whose canonical was hardcoded to /, so /usage and /playground declared themselves duplicates of the homepage. The body was an empty div, and there was no robots.txt or sitemap.

Each route is now prerendered to its own HTML file with its own head and rendered markup.

  • src/lib/seo.ts lists every page; src/lib/head.ts describes the head once, used at build time and on client navigation
  • bun run build adds an SSR build plus scripts/prerender.ts, writing one file per route and sitemap.xml
  • robots.txt, JSON-LD on the homepage, noindex on the 404
  • the catch-all rewrite is gone, so unknown URLs return a real 404 instead of the homepage at 200

Adding a route now requires adding it to PAGES, or it 404s on a direct hit.

Also adds CLAUDE.md, and moves the // comment out of overrides — npm reads every key there as a package name and refuses to run on it.

Summary by cubic

Prerenders each route to its own HTML and head so crawlers can index pages. Old: one index.html for every URL with canonical “/” and an empty body; New: each route serves rendered markup, its own canonical/title/OG/Twitter, emits a sitemap, and unknown URLs return a real 404 with noindex.

  • Build: apps/web runs vite build, vite build --ssr src/entry-server.tsx --outDir dist-ssr, then bun scripts/prerender.ts; outputs dist and dist-ssr. .gitignore, turbo.json, tsconfig.json, and knip.json updated.
  • Head/SEO: src/lib/seo.ts defines PAGES; src/lib/head.ts centralizes tags and the client applies them on SPA navigation. JSON‑LD on the homepage; robots.txt added; 404 serves noindex.
  • Deploy: Vercel catch‑all rewrite removed; cleanUrls: true, trailingSlash: false. Root vercel.json is the source of truth.
  • Required: When adding a route, add it to PAGES in apps/web/src/lib/seo.ts, or direct hits 404 and the page is excluded from the sitemap.
  • QA: Verify /robots.txt and /sitemap.xml load, /usage and /playground have correct canonicals/titles, and unknown paths return a 404 with noindex.
  • Tooling: Moved the comment out of overrides into overridesComment in the root package.json so npm can read the workspace.