NEXT: Home attachments upload as client-driven R2 multipart parts of any size

TE
7d ago
+2203-65

Transport only; the cap removal that makes it reachable is #27016. The 100 MB cap (validateAttachmentMeta, Rust MAX_ATTACHMENT_BYTES, the conformance corpus, Swift AttachmentMedia/HomeComposerCheck) stays in this PR, so the multipart path is not reachable above 100 MB until #27016 lands.

Takes back #26673: its inline abortMultipartUploads?.(prefix) in deletePrefix is replaced by the same call behind the typed abortsByPrefix guard, which slot expiry and the abort queue now share; behaviour is unchanged.

This PR alone is about 1,450 changed lines: about 760 for the transport and about 685 for the review fixes and their tests. It isn't split because the review fixes (part claims, completion recovery, GC abort) are what make the transport safe to land: the transport alone would ship the races the review found.

Part 2 of 2 of #26432 (R2 multipart transport for Home attachments; its session is gone, the attachments lane owns it now), with all of #26432's review findings fixed. Supersedes #26432.

Leo's standing order: "Attachments take arbitrarily large files of any type; there is no size cap" (Leo, 2026-10-26: "WHY THE FUCK SHOULD THERE BE A FUCKING SIZE CAP ANYWAYS ... I ALREADY SAID WE NEED ARBITARRILY LARGE FILES OF ANY TYPE").

What changes

  • Part plan: multipartPlan in home-core uses 32 MiB parts, growing in whole MiB only when a file would need more than 10,000 parts. 104 GiB is 3,328 × 32 MiB; 1 TiB is 9,987 × 105 MiB.
  • One request per part: the intent creates the R2 multipart upload. Each part is its own request (PUT /v1/home/attachments/multipart/<conv>/<slot>).
    • The part streams into its R2 part and into a resumable SHA-256. The owner keeps the hash state between parts, so no Worker invocation's subrequests grow with the file.
    • The last part must produce the declared sha256. Complete is one R2 call with the recorded etags.
    • This fixes finding 1: about 104 GiB used to pass the 10,000-subrequest limit.
  • Part numbers (finding 2): a part number must be an integer from 1 to the planned part count, never above 10,000, or the part gets 400 attachment.part_invalid. Complete checks its parts list too (attachment.parts_invalid).
  • GC (finding 3): each alarm aborts at most 100 due slots, oldest first, and re-arms while more are due. One failed abort is logged and skipped instead of stalling every retry on the same page.
  • Presign tests (finding 4): they now say they cover the older-client flow, and the no-S3 test checks both answers.
  • CmuxHomeCore: reads part_size and sends parts in order. The contract is unchanged.
  • Every size has a transport (coordinator finding 2): uploadModeFor picks the transport for each intent.
    • Up to 32 MB the file streams through the Worker, as before.
    • A client that sends parts (multipart: true, as CmuxHomeCore does) gets multipart at every size.
    • An older client gets a single request. With the S3 settings that is the presigned PUT, up to R2's single-PUT size. Without them it is the Worker stream, up to the zone's request body limit (HOME_ATTACHMENT_WORKER_BODY_BYTES, default 100 MB), which covers every size the server accepts today.
    • Before this, such an intent answered 503 attachment.large_unavailable past 32 MB.
    • Past what one request can carry, an older client gets 400 attachment.multipart_required, not a size refusal. The presigned path stays only where S3 exists.

Known limits:

  • Above about 931 GiB a part exceeds 100 MB, the request cap on the hosted Worker's Free and Pro plans (Business allows 200 MB, Enterprise 500 MB). Stacked #27015 sends such parts straight to R2 with presigned UploadPart URLs and verifies the sha256 by reading the object back in bounded steps.
  • R2's default rule clears incomplete uploads after 7 days.

Risk: normal (Home attachment upload path).

Red / green

Tests were committed first at f816eb18183 and were red without the transport: 7 failures across the three api attachment suites and home-core's attachments-multipart.test.ts. Green at f1247591a12:

  • api attachment suites: 29 pass.

  • home-core: 444 pass.

  • home-server: 11 pass.

  • tsc -p . (api) and the home-server typecheck are clean.

  • Finding 2: red at 605e62f88c9 (cd backend/apps/api && bun --bun x vitest run test/home-attachments-review.test.ts -t "S3 settings|every size": 2 failures, the legacy intent answered 503 and uploadModeFor did not exist). Green at af9585581fc: api attachment suites 30 pass, home-core 444, home-server 13.

  • #26673's head is merged in, now at 810fbee7cf5 through lane/attach-multipart-fs-bucket-gc (f144c4c0303).

  • Review BLOCK (Codex Astra), three findings, each red then green:

    • P1 completion errors. The red commit is b2b893db8fb (cd backend/apps/api && bun --bun run test -- test/home-attachments-multipart.test.ts: 2 failed). First, a transient R2 error (10001) or a dropped connection answered [400, attachment.size_mismatch] where [503, attachment.complete_unavailable, retryable] was expected. Second, a completion R2 made whose answer was lost answered 400 where [200, stored] was expected. The fix is b2415dbe869, which passes all 3 tests:
      • The error's code decides, never its message: R2Error.code, the v4 code the binding appends, or FsBucketError.code.
      • An integrity failure (InvalidPart, InvalidPartOrder, EntityTooSmall, BadDigest, BadUpload) still ends the slot.
      • On NoSuchUpload or a transient error, the Worker checks for the object at the declared size and the declared sha256. The upload now carries the sha256 as customMetadata.sha256. If the object is there, it is committed.
      • Otherwise the slot opens again and the client can complete again.
    • P1 concurrent part PUTs. The red commit is 4f9f06ee21d. Two gated PUTs of different bytes for part 1 make the completion answer [400] where [200, stored] was expected: the loser overwrote the R2 part. The fix is 29b33edb010, which passes all 5 tests:
      • claimMultipartPart claims the part in the ConversationDO before any byte reaches R2.
      • A second PUT gets 409 attachment.part_busy with retryable: true.
      • The record must name the claim. A part that was not recorded releases its claim at once. A claim its request never released lapses after the upload grace.
    • P2 GC of open uploads. The red commit is 0ed97184e66. In the api suite, uploadPart on a deleted conversation's R2 upload still resolved. In home-server, .multipart kept the slot's upload and an upload no slot names. The fix is 2082dcb89f2:
      • deleteAttachmentStorage aborts every multipart slot's upload before it forgets the slots.
      • On a bucket that can list its open uploads (FsBucket), the prefix delete and the delayed purge also abort every open upload under the prefix.
    • The fs-bucket pieces belong on #26673. They are on lane/attach-multipart-fs-bucket-gc, on top of #26673's 810fbee7cf5, and merged here at f144c4c0303:
    • At f144c4c0303: api attachment suites 8 files, 59 pass; home-server 24 pass, 0 fail; both typechecks clean.
    • Full api suite at 2082dcb89f2: 1,210 pass, 3 fail. The failures are chief-brain-spawn, home-group-large-e2e (a 120 s timeout) and chief-github. All 11 tests in those files pass when they run alone, so these are load flakes in the parallel run, not this change.

  • Second review BLOCK (at f144c4c0303), two findings, each red then green. The command is cd backend/apps/api && bun --bun run test -- test/home-attachments-multipart.test.ts:

    • P2 owner commit lost after R2 completed. The red commit is 807ac09ded9: 2 failed.
      • "the owner's retry verifies the joined object and commits it; another participant still gets 403": the retry answered [403] where [200, stored] was expected.
      • "two retries that both find the joined object commit it once, and the later one never deletes it": the gated retry answered 403 before it reached the commit.
    • The fix is 99f1f4a8a1f:
      • completingSlot lets the slot's own actor complete again an uploading multipart slot, before the expiry grace. A non-owner still gets 403 attachment.slot_invalid.
      • The join finds the object by its size and sha256.
      • commitAttachment takes the slot's object id, so a retry whose slot was already committed answers stored and never deletes the kept object.
    • P2 claim timeout vs slot lease. The red commit is 03c6382452a. "before the claim times out the part is busy; after it, the retry takes the part, the vanished request can no longer write it, and the upload completes" runs on an injected clock (vi.setSystemTime, no sleeps). Just before the claim's timeout the retry answered [403, attachment.slot_invalid] where [409, attachment.part_busy] was expected.
    • The fix is 97db95a9c8e, which passes all 9 tests:
      • PART_CLAIM_MS is 5 min, under the 15-min lease the claim gives the slot.
      • A live request renews its claim every 2 min while bytes arrive. Each renewal extends the slot's lease by at most one lease.
      • A request whose claim lapsed, or could not be renewed, stops before R2 keeps its bytes.
    • At 97db95a9c8e: api attachment suites 8 files, 62 pass; home-server 24 pass, 0 fail; both typechecks and the prepush gate clean; fs-bucket.ts unchanged.

At af9585581fc the full api suite had 2 failures (inference-free-routes, inference-engine-auth), which fail the same way on base.

  • Not run: the CmuxHomeCore Swift tests (WireHomeSourceTests). These need a Mac. CmuxHomeCore was not changed for the two new retryable answers (attachment.complete_unavailable, attachment.part_busy). HomeWireClient.check maps them to .indeterminate. The client has no retry of just that part or just the completion yet.

  • Third review BLOCK (Codex Sol, at 4550ee54e30), two P2s, red then green:

    • Untracked upload after a failed intent. If the owner call that records the new R2 upload failed or lost its answer, the upload stayed open with no slot naming it. The intent now aborts it.
    • Lost upload id after a failed expiry abort. A transient abort failure dropped the slot and with it the only upload id. Now any failure other than NoSuchUpload queues the id in home_attachment_aborts, in the same step that forgets the slot. Each wake retries due aborts with the drop backoff and dead letter.
    • The red commit is 819c3f9bc78 (cd backend/apps/api && bun --bun x vitest run test/home-attachments-multipart.test.ts -t "never left without a record": 2 failed). The fix is 37585a9e140: api attachment suites 9 files, 66 pass; home-core 444; home-server 33; api typecheck clean.

  • Fourth review BLOCK (Codex Astra, at 37585a9e140), two P2s, red then green:

    • Orphan whose abort fails too. If the slot could no longer take the new upload, or the owner call failed, and the Worker's abort also failed, nothing recorded the upload. setMultipartUpload now takes the object key, and a slot that can't take the upload queues it in home_attachment_aborts at once. If the owner call fails and the abort fails, the Worker asks the owner once more (queueMultipartAbort). The key is a trailing optional argument, so an older caller still works.
    • Self-hosted completion retry. FsBucket.resumeMultipartUpload throws NoSuchUpload synchronously for an upload a completion already closed. That escaped as a 500 before completedObject could find the joined object. The resume now runs inside the try.
    • The red commit is 703593d3dfc (3 failed); the fix is 5b753ed6d9a.

  • Fifth review BLOCK (Codex Astra, at 5b753ed6d9a), one P2, red then green:

    • Transient part failure. An uploadPart rejection from R2 or the network answered 400 attachment.size_mismatch, which the client treats as final. Now only the body's own fault (wrong length, a lapsed claim) is a refusal. Anything else answers 503 attachment.part_unavailable, retryable: true, and the same part can be sent again.
    • The red commit is d144d9f2370 (400 where 503 was expected); the fix is 306f3ab50cc.
    • At 306f3ab50cc: api attachment suites 70 pass; home-core 444; home-server 33; api typecheck clean.

  • Sixth review BLOCK (Codex Astra, at 306f3ab50cc), two P2s and a P3, red then green:

    • Claim lapse while R2 writes the part. The claim renewed only while bytes arrived, so a slow uploadPart answer let another request take a part this one might still write. A heartbeat now renews the claim every PART_RENEW_MS until uploadPart settles. It ticks once a second and is cleared in finally.
    • Lost part acknowledgment. A retry of a part already recorded got 409 attachment.part_order. It now answers the stored part number, length and etag without reaching R2, so the client goes on to the next part.
    • P3: CmuxHomeCore reads part_size with the exact-integer conversion, so a bad number is attachment.bad_answer, not a trap.
    • The red commit is f552137abb9 (2 failed: [200] where [409, part_busy] was expected, and [409] where [200, 1, 33554432] was expected). The fix is cb32796a6d1, and 156ee772775 updates the older replay assertion to the stored answer and no R2 call.
    • At 156ee772775: api attachment suites 72 pass; home-core 444; home-server 33; api typecheck clean.

  • Seventh review BLOCK (Codex Astra, at 156ee772775), two P2s, red then green:

    • Stale last part. A request whose claim was taken while R2 wrote its part could still end the slot on a hash mismatch and abort the upload under the newer holder. It now stops before it touches the slot, and the last part confirms its claim before a mismatch ends the slot.
    • Claim loss not retryable. Claim-loss and not-recorded answers (409 attachment.part_order) now carry retryable: true. The retry gets the stored part, or 403 when the slot ended.
    • The red commit is a67890f9b2b ([400] where [409, part_order, retryable] was expected); the fix is 0b3ce1e57a0.
    • At 0b3ce1e57a0: api attachment suites 73 pass; home-core 444; home-server 33; api typecheck clean.

  • Eighth review BLOCK (Codex Astra, at 0b3ce1e57a0), a P1 and a P2:

    • P1, stale part write landing last. The concern was a request whose claim lapsed while the owner was unreachable, writing its part after a newer request recorded it. The tests show R2 joins only parts whose recorded etags match, so any stale write is refused at completion (400, the slot ends) and never committed as the declared file. That holds for other bytes and for the same bytes alike: the local R2 gives each part write its own etag.
    • The remaining cost: when the owner is unreachable for longer than a claim's lifetime (5 min) while R2 writes a part, the client starts that file again. Nothing is corrupted or lost.
    • P2, closed self-hosted upload on a part PUT. FsBucket.resumeMultipartUpload throws NoSuchUpload synchronously; the part handler now answers 403 attachment.slot_invalid (503 retryable for other failures) instead of a 500.
    • The red/probe commit is 53c6929434 (the closed-upload test failed with an uncaught NoSuchUpload); the fix is 1019cf84795.
    • At 1019cf84795: api attachment suites 76 pass; home-core 444; home-server 33; api typecheck clean.

  • Ninth review BLOCK (Codex Sol, at 1019cf84795), two P2s, red then green:

    • Owner unreachable for everything. The owner could be unreachable for the record, the abort and the abort queue alike. The slot itself is now the record: an expired multipart slot with no upload id aborts every open upload under its own key, on a bucket that can list them (FsBucket). A failed prefix abort is queued with the drop backoff. R2's binding cannot list uploads, and its lifecycle rule ends them.
    • Expired slot. setMultipartUpload refuses a slot past its time and queues the upload for an abort.
    • The red commit is 58377e6e304 (2 failed); the fix is f8a3e19a5ac.
    • At f8a3e19a5ac: api attachment suites 78 pass; home-core 444; home-server 33; api typecheck clean.

  • Tenth review BLOCK (Codex Astra, at f8a3e19a5ac), two P2s, red then green:

    • Swift short reads. FileHandle.read(upToCount:) may return fewer bytes than asked before the end. CmuxHomeCore now fills each part (WireHomeSource.readPart) before sending it. The test is aMultipartPartIsReadWholeAcrossShortReads.
    • Leftover part row. A part row left without its slot's hash state blocked that part with a primary-key error. recordMultipartPart now upserts it. The red commit is 307f730d27c (UNIQUE constraint failed).
    • The fix is f4270164669; the Swift test stub was corrected at 6cec785b07f.
    • At f4270164669: api attachment suites 79 pass; home-core 444; home-server 33; api typecheck clean.

  • Eleventh review BLOCK (Codex Astra, at 6cec785b07f), one P2, red then green:

    • Concurrent completion retries. A retry that failed transiently could reopen the slot while another completion was still in R2. That completion's commit was then refused, and finish deleted the completed object.
    • commitAttachment now takes a multipart slot that is open as well as one that is uploading. It is only called after R2 joined the parts and the object was checked.
    • The red commit is 5455edff310 ([403] where [200, stored] was expected); the fix is b352608d655.
    • At b352608d655: api attachment suites 80 pass; home-core 444; home-server 33; api typecheck clean.

  • Twelfth review BLOCK (Codex Astra, at b352608d655), one P2, red then green:

    • Reservation kept after a failed intent. When R2 could not create the upload, or the owner call failed and the Worker's abort succeeded, the slot and its quota reservation stayed about 75 minutes.
    • The intent now drops the open multipart slot (dropUploadSlot) and refunds the reservation at once. A live upload (its abort failed) keeps the slot as its record until expiry aborts it.
    • The red commit is bb63be5e141 (2 failed); the fix is 9152380633a.
    • At 9152380633a: api attachment suites 82 pass; home-core 444; home-server 33; api typecheck clean.

  • Thirteenth review BLOCK (Codex Astra, at 9152380633a), one P2 in round 12's fix, red then green:

    • Slot dropped before its refund. The release dropped the slot before refunding and swallowed a refund failure, so nothing would retry the refund. Now the refund comes first, and the slot is dropped only once it succeeded; otherwise the slot's expiry refunds.
    • The red commit is 139861736a0; the fix is d098a3f57c2.
    • At d098a3f57c2: api attachment suites 83 pass; home-core 444; home-server 33; api typecheck clean.
    • CmuxHomeCore suites (WireHomeSourceTests, AttachmentTests, AttachmentRoundTwoTests, AttachmentAnyCountTests, AnyFileAttachmentTests) passed on a fleet Mac at 4550ee54e30; pass again on a fleet Mac at 6cec785b07f (aws-m4pro-3).

Changelog

Fixed: Home attachments of any size upload in parts instead of failing past the Worker's limits.

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Home attachment uploads over 32 MB now use client-driven R2 multipart transfers instead of a single presigned PUT, so arbitrarily large files no longer fail when a Worker invocation's subrequests pass the 10,000 limit. The client-side 100 MB cap is removed: any size is accepted and streamed.

  • The owner plans 32 MiB parts (grown in whole MiB only past 10,000 parts) and each part is its own PUT /v1/home/attachments/multipart/<conv>/<slot> request that streams into its R2 part and into a resumable SHA-256 carried between requests; completion commits from the recorded etags.
  • Part numbers are validated as integers from 1 to the planned count (never over 10,000), with 400 attachment.part_invalid/attachment.parts_invalid for bad values; completion also checks the client's parts list against the stored etags.
  • Attachment GC now expires at most 100 due slots per alarm, oldest first, re-arming while more are due; one failed abort is logged and skipped.
  • Older clients that do not send multipart: true still get one request: a presigned PUT where S3 exists, otherwise the Worker stream up to the zone's body limit (new HOME_ATTACHMENT_WORKER_BODY_BYTES env, default 100 MB); past what one request can carry, the intent answers 400 attachment.multipart_required instead of 503.
  • The self-hosted bucket (home-server fs-bucket) streams multipart parts to files and assembles the object on complete without holding it in memory; resumed uploads whose id is forged or minted for another key answer NoSuchUpload, and completion validates the parts list before writing.

Known limits: a part exceeds 100 MB above about 931 GiB (the hosted Worker's Free/Pro request cap); R2's default rule clears incomplete uploads after 7 days.