feat(host-service): sweep the attachment store by last use, and keep fileId on cloud copies

Closed
SU
15d ago
+510-33

Summary

PR 3 of the attachments plan (v6): host-service + shared. It covers seam 8 (the limits module), seam 2 (retention of the host store by last use), and the fileId carry-through of seam 3. It does not depend on #8704 (which now includes the API change from #8712). This PR uses only attachment.resolve, which is on main.

Why

  • The host store never forgets. attachments.upload / importFromCloud write into $HOST_MANIFEST_DIR/attachments/<id>/, and nothing removes those files. There is no session delete to hook: chat-v3 only has closeSession, nothing deletes chat.db rows, and a terminal launch has no session.
  • The first prompt of a cloud session loses its identity. A cloud launch imports its attachments from storage at first boot, and the host builds the first-prompt parts from that store record, so the part has only a host attachmentId. That is the one kind of attachment that a reader without the box (web, a shared view, a resumed session on a recreated box) cannot render.
  • Limits are spread out. MAX_INLINE_ATTACHMENT_BYTES was a host-local constant, and the per-send 10 was hard-coded.

How it works

  • lastUsedAt in metadata.json. Upload and import set it. Every path that gives a file to a client or an agent goes through the new handOutAttachment, which sets it again: attachments.read, the # Attached files block of agents.run, and the ACP resolver in chat-v3/acpHarnesses.ts.
  • sweepAttachmentStore (attachments/sweep.ts) runs once at startup and then daily, beside the project purge. It does not run in a sandbox: a box's store is only a cache that dies with the box. It removes entries whose lastUsedAt is older than ATTACHMENT_RETENTION_DAYS (30). An entry written before this change has no stamp. The sweep stamps it, so it gets the full 30 days. For each entry, the metadata check and a renameSync to .sweeping-<id> run in the same tick, so a hand-out on this event loop cannot slip between them. The recursive delete then runs async (fs/promises.rm), off the main loop. A leftover .sweeping-* from a crash is removed on the next pass.
  • Lookup by fileId (for seam 4). importCloudAttachments writes fileId into the metadata and into a one-line index file .by-file-id/<fileId> → attachmentId. handOutAttachmentByFileId reads it in O(1), checks it against the metadata, and stamps the entry. Delete and the sweep remove the index entry with the store entry.
  • First-prompt parts carry fileId. chatLaunchTarget spreads fileId (and sizeBytes) from the store metadata into each part. The sandbox self-seed already sends the imported ids through it, so a cloud launch's first prompt now names its storage object. importFromCloud's result also gains fileId (an additive output field).
  • Protocol. userContentSchema's attachment part gains optional fileId (uuid) and sizeBytes. These are additive. attachmentId is still required. Making it optional is the next PR, together with the resolver that can serve a part without it.
  • Limits. @superset/shared/attachment-limits gains MAX_INLINE_ATTACHMENT_BYTES (moved from the host's constants.ts, which is deleted; the attachments index still re-exports it), MAX_THUMBNAIL_BYTES, and ATTACHMENT_RETENTION_DAYS. importFromCloud, materializeIntoWorkspace and cloudAgentLaunch.attachmentFileIds use MAX_ATTACHMENTS (still 10, so their inputs are unchanged).

Where this differs from the plan, or the plan is out of date against main

  • A first-prompt part carries both attachmentId and fileId. The plan says "exactly one of the two is set". On main today, every adapter and every released mobile build reads attachmentId, so a fileId-only part would reach the agent as "could not be read" until the resolver from the next PR ships. Readers still branch on fileId first. The next PR can drop attachmentId from host-built parts once its resolver is in place.
  • ATTACHMENT_PARTS_MIN_HOST_VERSION is not here. It is "the first host release carrying seam 4", so it belongs in the next PR.
  • MAX_THUMBNAIL_BYTES is defined but not used yet. Desktop's useChatImageSource keeps its local copy until the desktop PR switches it over. That keeps this PR to host-service + shared.
  • Not changed: the two .max(10) in packages/trpc (cloudWorkspace.create's attachmentFileIds, attachFiles). They are API code, outside this PR.
  • Line references are out of date. agents.ts:737 is now sessions/create-chat-session.ts (createChatSession), which spreads chatLaunchTarget's attachments into the prompt.

tRPC compatibility

  • attachments.importFromCloud: the output gains fileId, which is additive. The input is unchanged (max(MAX_ATTACHMENTS) = max(10)).
  • attachments.materializeIntoWorkspace: the input is unchanged, for the same reason.
  • attachments.read, upload, delete: the contracts are unchanged. read now also writes lastUsedAt.
  • The chat protocol's attachment part: two new optional fields. Older hosts accept them, because the part is a looseObject.
  • No new host procedure, so no client guard and no MIN_HOST_SERVICE_VERSION change.

Testing

  • sweep.test.ts: the sweep removes an entry unused past retention and keeps a recent one; an unstamped entry is stamped, not removed; an entry handed out after the pass started is kept; the fileId index is removed with its entry; handOutAttachment / handOutAttachmentByFileId stamp what they hand out.
  • attachments.test.ts: read stamps the entry; importCloudAttachments records the fileId, and a lookup by that fileId finds the copy (with a local Bun.serve standing in for the presigned GET).
  • agents.test.ts: a cloud launch's first-prompt part carries the fileId it was imported from (import, then chatLaunchTarget); the agents.run attached files block (buildTerminalAgentLaunch) stamps what it lists.
  • bun run typecheck passes in packages/shared, packages/chat, packages/chat-runtime, packages/host-service and apps/desktop. Biome is clean on the changed files.
  • packages/host-service bun test: 2644 pass, 1 fail. The failure is health.check > serialises the current boot's stamps … in sandbox mode. It reads the real boot stamps of the machine it runs on, and this workspace is a real cloud box, so ready is true and bundle is set. It does not touch attachment code. no-main-loop-blocking.test.ts passes, and the rmSync count in storage.ts is still 2.
  • packages/shared (1444) and packages/chat (94): all pass.
  • apps/mobile typecheck has 190 errors both with and without this change (checked on a clean tree). None of them are in attachment code.
  • I did not run a desktop, mobile, or cloud box end to end.

Known limitations

  • An attachment directory that has no metadata.json (a download that failed before its metadata was written) is skipped by the sweep, not removed.
  • A recreated box imports a file again on resume and caches it twice. The plan accepts this: the index points to the newest copy, and the sweep removes the rest.
  • The plan's "store path outside the worktree" fixture for Codex and Claude goes in the next PR, because the adapters are what read those paths.