feat(host-service): sweep the attachment store by last use, and keep fileId on cloud copies
Closed
SU
15d ago+510+510+012345678901234567890123456789-33-33-01234567890123456789
Summary
PR 3 of the attachments plan (v6): host-service + shared. It covers seam 8 (the limits module), seam 2 (retention of the host store by last use), and the fileId carry-through of seam 3. It does not depend on #8704 (which now includes the API change from #8712). This PR uses only attachment.resolve, which is on main.
Why
- The host store never forgets.
attachments.upload/importFromCloudwrite into$HOST_MANIFEST_DIR/attachments/<id>/, and nothing removes those files. There is no session delete to hook: chat-v3 only hascloseSession, nothing deletes chat.db rows, and a terminal launch has no session. - The first prompt of a cloud session loses its identity. A cloud launch imports its attachments from storage at first boot, and the host builds the first-prompt parts from that store record, so the part has only a host
attachmentId. That is the one kind of attachment that a reader without the box (web, a shared view, a resumed session on a recreated box) cannot render. - Limits are spread out.
MAX_INLINE_ATTACHMENT_BYTESwas a host-local constant, and the per-send10was hard-coded.
How it works
lastUsedAtinmetadata.json. Upload and import set it. Every path that gives a file to a client or an agent goes through the newhandOutAttachment, which sets it again:attachments.read, the# Attached filesblock ofagents.run, and the ACP resolver inchat-v3/acpHarnesses.ts.sweepAttachmentStore(attachments/sweep.ts) runs once at startup and then daily, beside the project purge. It does not run in a sandbox: a box's store is only a cache that dies with the box. It removes entries whoselastUsedAtis older thanATTACHMENT_RETENTION_DAYS(30). An entry written before this change has no stamp. The sweep stamps it, so it gets the full 30 days. For each entry, the metadata check and arenameSyncto.sweeping-<id>run in the same tick, so a hand-out on this event loop cannot slip between them. The recursive delete then runs async (fs/promises.rm), off the main loop. A leftover.sweeping-*from a crash is removed on the next pass.- Lookup by fileId (for seam 4).
importCloudAttachmentswritesfileIdinto the metadata and into a one-line index file.by-file-id/<fileId>→ attachmentId.handOutAttachmentByFileIdreads it in O(1), checks it against the metadata, and stamps the entry. Delete and the sweep remove the index entry with the store entry. - First-prompt parts carry
fileId.chatLaunchTargetspreadsfileId(andsizeBytes) from the store metadata into each part. The sandbox self-seed already sends the imported ids through it, so a cloud launch's first prompt now names its storage object.importFromCloud's result also gainsfileId(an additive output field). - Protocol.
userContentSchema's attachment part gains optionalfileId(uuid) andsizeBytes. These are additive.attachmentIdis still required. Making it optional is the next PR, together with the resolver that can serve a part without it. - Limits.
@superset/shared/attachment-limitsgainsMAX_INLINE_ATTACHMENT_BYTES(moved from the host'sconstants.ts, which is deleted; the attachments index still re-exports it),MAX_THUMBNAIL_BYTES, andATTACHMENT_RETENTION_DAYS.importFromCloud,materializeIntoWorkspaceandcloudAgentLaunch.attachmentFileIdsuseMAX_ATTACHMENTS(still 10, so their inputs are unchanged).
Where this differs from the plan, or the plan is out of date against main
- A first-prompt part carries both
attachmentIdandfileId. The plan says "exactly one of the two is set". On main today, every adapter and every released mobile build readsattachmentId, so a fileId-only part would reach the agent as "could not be read" until the resolver from the next PR ships. Readers still branch onfileIdfirst. The next PR can dropattachmentIdfrom host-built parts once its resolver is in place. ATTACHMENT_PARTS_MIN_HOST_VERSIONis not here. It is "the first host release carrying seam 4", so it belongs in the next PR.MAX_THUMBNAIL_BYTESis defined but not used yet. Desktop'suseChatImageSourcekeeps its local copy until the desktop PR switches it over. That keeps this PR to host-service + shared.- Not changed: the two
.max(10)inpackages/trpc(cloudWorkspace.create'sattachmentFileIds,attachFiles). They are API code, outside this PR. - Line references are out of date.
agents.ts:737is nowsessions/create-chat-session.ts(createChatSession), which spreadschatLaunchTarget's attachments into the prompt.
tRPC compatibility
attachments.importFromCloud: the output gainsfileId, which is additive. The input is unchanged (max(MAX_ATTACHMENTS)=max(10)).attachments.materializeIntoWorkspace: the input is unchanged, for the same reason.attachments.read,upload,delete: the contracts are unchanged.readnow also writeslastUsedAt.- The chat protocol's attachment part: two new optional fields. Older hosts accept them, because the part is a
looseObject. - No new host procedure, so no client guard and no
MIN_HOST_SERVICE_VERSIONchange.
Testing
sweep.test.ts: the sweep removes an entry unused past retention and keeps a recent one; an unstamped entry is stamped, not removed; an entry handed out after the pass started is kept; the fileId index is removed with its entry;handOutAttachment/handOutAttachmentByFileIdstamp what they hand out.attachments.test.ts:readstamps the entry;importCloudAttachmentsrecords the fileId, and a lookup by that fileId finds the copy (with a localBun.servestanding in for the presigned GET).agents.test.ts: a cloud launch's first-prompt part carries the fileId it was imported from (import, thenchatLaunchTarget); theagents.runattached files block (buildTerminalAgentLaunch) stamps what it lists.bun run typecheckpasses inpackages/shared,packages/chat,packages/chat-runtime,packages/host-serviceandapps/desktop. Biome is clean on the changed files.packages/host-servicebun test: 2644 pass, 1 fail. The failure ishealth.check > serialises the current boot's stamps … in sandbox mode. It reads the real boot stamps of the machine it runs on, and this workspace is a real cloud box, soreadyis true andbundleis set. It does not touch attachment code.no-main-loop-blocking.test.tspasses, and thermSynccount instorage.tsis still 2.packages/shared(1444) andpackages/chat(94): all pass.apps/mobiletypecheck has 190 errors both with and without this change (checked on a clean tree). None of them are in attachment code.- I did not run a desktop, mobile, or cloud box end to end.
Known limitations
- An attachment directory that has no
metadata.json(a download that failed before its metadata was written) is skipped by the sweep, not removed. - A recreated box imports a file again on resume and caches it twice. The plan accepts this: the index points to the newest copy, and the sweep removes the rest.
- The plan's "store path outside the worktree" fixture for Codex and Claude goes in the next PR, because the adapters are what read those paths.