feat(files): the files layer — schema, router, media route (SUPER-2077)

Merged
SA
2mo ago
+10034-37

Rebase + bot review triage

Rebased onto main after #6954's squash-merge: absorbed the apps/usercontent rename (media host ported by hand — git kept both directories), the frame. hostname, main's publish/delete/manifest review fixes, and regenerated the migration as 0101_add_files_and_attachments on the merged schema.

CodeRabbit findings fixed: page delete now removes page_version attachment rows and orphaned files (rows + bytes; no FK carries the cascade); the pending-file sweep claims rows before deleting objects and complete's ready-flip is guarded on pending (no more ready-with-no-bytes race); ticketed media responses cache only until their ticket expires, mirroring the thumbnail rule.

Dismissed: full-content SHA-256 verification at complete — the ratified verification design is size + magic-byte sniff (decision 8); the client-supplied hash only feeds that client's own republish dedup, so a false hash poisons nobody else, and hashing 1 GiB server-side per upload buys nothing for that.

Links

Summary

  • The files layer ships as architecture for v1: files + polymorphic attachments tables, a file router (createUpload / complete / sign / list / delete), and media.supersetusercontent.com on the content Worker with Range support — no UI on purpose; issues/chat attachment UI comes later, page assets (SUPER-2078) stand on this.

Why / Context

v1 ships video inside pages and architectural support for attachments everywhere else. Bytes never pass through the API: uploads go direct to R2 by presigned PUT, serving goes through the Worker with tickets, and the API only decides who may.

How It Works

  • Upload: createUpload records a pending row and returns a presigned PUT whose signature covers the declared content type and length (first size gate, 1 GiB single PUT; multipart when the cap is hit in practice).
  • Verification: complete is the gate a presigned PUT cannot be — HEAD confirms the size createUpload was told, a ranged read of the first 8 KB sniffs the real type (magic bytes; anything that smells like a scriptable document is named one), and only then is the file ready. A size mismatch deletes row and object.
  • Serving: /files/<fileId>[/<filename>]?ticket=… on media.. The day-rounded file ticket carries the sniffed content type, so the Worker serves with no database. Serve-time policy: HTML/XML always download, SVG renders only as a subresource image (Sec-Fetch-Dest), PDF/media/plain text inline — all under nosniff + Content-Security-Policy: sandbox; default-src 'none', Superset-Storage-Key, private, max-age=86400, immutable, and real 206s for Range so video seeks.
  • Access always derives from a readable parent through attachments (page_version implemented; issue/doc/chat_session/comment refuse loudly with NOT_IMPLEMENTED until their surfaces ship), or from the uploader before attachment.
  • Sweep: pending rows older than a day are dropped — objects first, so a re-run repairs a half-finished sweep — via a QStash-verified job at /api/files/jobs/sweep.

Manual QA Checklist

No UI ships; the deployed origin is blocked on SUPER-2080. The Worker and router logic were exercised locally (Testing).

  • After deploy: upload via CLI-to-be (SUPER-2078) round-trips create → PUT → complete → sign
  • A video attachment seeks in <video> from media.
  • An HTML file uploaded as image/png downloads instead of rendering

Testing

  • bun run typecheck — @superset/db, shared, trpc, content, api
  • bun test — shared 881 (incl. new file-ticket + policy tests), trpc file 5 (sniff) + page 50, db 13
  • Migration 0100_add_files_and_attachments generated by drizzle-kit (never applied locally); applies through the deploy flow
  • Worker, locally against wrangler's simulated R2: full media matrix — inline video 200 with length; filename suffix names the download; Range explicit/open-ended/suffix → correct 206 + Content-Range; past-the-end → 416; html ticket downloads; svg inline as Sec-Fetch-Dest: image, download on navigation; missing/wrong-file/page tickets → 404; the pages origin refuses /files/*; CSP-sandbox, storage-key, day cache headers present

Design Decisions

  • Sniffed type in the ticket, not a Worker DB lookup: the API signs what it verified; serving survives an API outage and metadata stays immutable.
  • Plain terse TRPCErrors, not userError i18n keys: nothing user-facing renders these yet; they get keys (and sixteen translations) when the attachments UI ships. Deliberate deviation, matching several existing publish errors.
  • Enum for parent kinds: a new kind is a one-line ALTER TYPE, and an unknown kind can never be inserted.

Known Limitations

  • No attach/detach procedures yet — parents attach in their own writes (directory publish does it in SUPER-2078's transaction).
  • The sweep job needs its QStash schedule created (added to SUPER-2080).
  • MEDIA_URL env is new: production vars + MEDIA_URL_DEV for previews to create alongside the rest.

Follow-ups

  • SUPER-2078: directory publish + per-version manifest assets + the ticketed asset door.
  • Desktop main-process uploader lands with its first UI consumer (decision 7).

Summary by cubic

Adds the files layer — files and polymorphic attachments schema, a file router, and media.supersetusercontent.com on the usercontent Worker — so issues, chat, and page assets can attach and serve files without any UI yet. Bytes never touch the API: uploads presign directly to R2, serving is ticket-gated with Range support, and the new MEDIA_URL env is required.

  • createUpload records a pending row and returns a presigned PUT covering the declared content type and size (1 GiB cap).
  • complete HEAD-checks the size and sniffs the real type from the first 8 KB; a mismatch deletes row and object, and a scriptable document is named as one.
  • The serve-time policy keys on the sniffed type in the ticketed URL: HTML/XML download, SVG renders only as a subresource image, PDF/media/plain text inline, all under nosniff and a sandbox CSP, with browser cache capped at ticket life.
  • Access derives from a readable parent via attachments; page_version works, other parent kinds throw NOT_IMPLEMENTED.
  • A QStash-verified sweep drops pending rows older than a day (guarded so a late complete can't flip a swept row), objects first; page delete removes attachment rows and deletes files that no other attachment references.

Migration

  • The Worker is now apps/usercontent, serving pages (frame.supersetusercontent.com) and files (media.supersetusercontent.com); apps/content is gone.
  • Migration 0101 adds files + attachments; the merged origin work's 0100 drops chat_attachments and renames page_versions.blob_pathname to storage_key.
  • Set MEDIA_URL in production and MEDIA_URL_DEV for previews; the sweep's QStash schedule lands in SUPER-2080.

Summary by CodeRabbit

  • New Features
    • Added file uploads and attachments for pages and supported content.
    • Users can list, securely download, preview, and delete attached files.
    • Added ticketed media access with range requests and safe content handling.
  • Bug Fixes
    • Incomplete uploads older than 24 hours are automatically removed.
    • Improved cleanup of files left behind after page deletion.
  • Chores
    • Added media hosting configuration for local, preview, and production environments.