NEXT: Home attachment parts over the Worker's request limit go straight to R2, verified in bounded steps

Merged
TE
7d ago
+1025-375

Leo's standing order: "Attachments take arbitrarily large files of any type; there is no size cap".

Finding 1 on #26674 (merged at d6099db74c): a part over the Worker's request body limit (100 MB on Cloudflare Free and Pro) can't reach the Worker. That happens for files over about 931 GiB, where multipartPlan grows parts past 100 MB.

What changes

  • Choosing the transport. uploadModeFor answers direct when a multipart file's parts exceed the zone's request limit. The limit is HOME_ATTACHMENT_WORKER_BODY_BYTES, default 100 MB. direct needs the S3 settings; without them the intent answers 503 attachment.large_unavailable, because nothing else can carry those bytes. Parts the Worker can carry still go through it.
  • Intent answer. {mode: "multipart", transport: "direct", upload_url, part_size, part_count, slot}. Through-Worker answers now carry transport: "worker", which is additive.
  • Part URLs. POST upload_url with x-cmux-part-number: N returns {part_number, method: "PUT", url, headers: {content-length}, expires_at}.
    • url is a presigned R2 UploadPart URL. r2-presign.ts now signs the URL's own partNumber and uploadId.
    • The client PUTs the part there with no auth header and keeps R2's ETag response header, quoted or bare.
    • Each POST renews the slot's idle lease. Parts may go in any order.
  • Completion. POST multipart/complete {conversation, slot, parts} (parts is required for direct):
    • The first call joins the parts in one R2 call. Wrong etags get 409 attachment.part_missing, and the upload stays so the client can retry.
    • Each call then hashes the next 256 MiB of the joined object onto the SHA-256 state the owner keeps, and answers {state: "verifying", verified_bytes, byte_count}.
    • The client repeats the same POST. The last step answers the record, or, if the hash differs, 400 attachment.hash_mismatch and the object is deleted.
  • Bounded per call: every call makes the same R2 calls whatever the file's size (the join, then one range read), so #26432's finding 1 stays fixed.

Why the server reads the file back instead of using per-part checksums

  • R2's per-part x-amz-checksum-sha256 only proves each part against a hash the client also declares.
  • The composite R2 keeps is a hash of the part hashes, never the file's SHA-256.
  • Attachments are keyed and deduplicated by the file's SHA-256, so it has to be computed from the stored bytes.
  • The cost is one more read of the file, plus about 3 s of CPU per 256 MiB step after the upload. Only parts the Worker can't carry pay it, so direct is used only past the limit, not for every part whenever S3 is configured. Through-Worker parts are hashed inline with no extra read.

Client contract (CmuxHomeCore, Not compiled here)

  • putMultipart reads transport. For "direct" it POSTs for each part's URL, PUTs the part there, and keeps the ETag. HomeWireHTTPReply gained etag, which URLSessionHomeWireTransport sets from the response header.
  • upload repeats the completion while it answers "verifying".
  • Unchanged for transport: "worker", which covers every file up to about 931 GiB on the hosted defaults.

Red / green

  • Red at 88d7ac40c82: cd backend/apps/api && bun --bun x vitest run test/home-attachments-direct.test.ts gave 3 failures.
  • Green at 28ae00c3539:
    • The same command: 3 pass.
    • The api attachment suites (home-attachments, -review, -sha256, -direct): 33 pass.
    • home-core: 444 pass. home-server: 13 pass.
    • tsc -p . (api) and the home-server typecheck are clean.

Observed / Not run

  • Not run: the CmuxHomeCore Swift tests, including the new directPartsGoToTheirPresignedURLsAndTheCompletionRepeatsWhileVerifying. They need a Mac compile.
  • Not run against real R2: the tests land each part on the bucket binding with the uploadId from the presigned URL, because tests have no S3 endpoint. That an uploadId made by the binding takes S3 UploadPart is from R2's docs, not observed here.

Risk: normal (Home attachment upload path, only for parts over the zone's request limit).

After #26674 landed

  • Merged feat-cmux-next (with #26674 at d6099db74c) at b7c3ffb6045. The conflicted files take #26674 as landed, and this PR's own change (4550ee54..44cda719) is applied on top. In CmuxHomeCore, both the short-read readPart and the direct putDirectPart are kept.

  • At b7c3ffb6045: CmuxHomeCore WireHomeSourceTests, AttachmentTests, AttachmentRoundTwoTests, AttachmentAnyCountTests and AnyFileAttachmentTests pass on a fleet Mac (aws-m4pro-2). No Swift changed after it.

  • Review BLOCK (Codex Astra, at b7c3ffb6045), two P2s and a P3, red then green:

    • Lost direct commit. A direct completion whose owner commit failed could not be resumed (completingSlot only looked for multipart slots), so the object waited for expiry. The caller's uploading direct slot is now found too, and the last step reuses it. A commit that took effect is kept and a retry gets 403, as for multipart.
    • Join errors. A transient R2 join error answered non-retryable 409 part_missing, and a failing head() was a 500. Now R2's integrity codes keep 409; anything else answers 503 complete_unavailable, retryable.
    • P3: dropUploadSlot drops direct slots too.
    • The red commit is 13d3ff7055d (3 failed); the fix is f2337336b03, with the commit-took-effect expectation aligned at 384ee6d7436.
    • At 384ee6d7436: api attachment suites 89 pass; home-core 444; home-server 33; api typecheck clean.

  • Second review BLOCK (Codex Astra, at 384ee6d7436), two P2s:

    • Read-back failure. A failing R2 get or a broken body stream in a read-back step was a 500. It now answers 503 attachment.verify_retry, retryable, with the hash state unchanged. The red commit is f18686246a6.
    • Direct parts in memory. CmuxHomeCore loaded each direct part (hundreds of MiB or more) into memory. Each part is now copied to its own temporary file 8 MiB at a time (WireHomeSource.partFile) and sent with upload(fromFile:), which URLSession streams. The file is removed once its upload returns. The test is aDirectPartIsCopiedToItsOwnFileInBoundedChunks.
    • The fix is bd57de5f544. At bd57de5f544: api attachment suites 90 pass; home-core 444; home-server 33; api typecheck clean. The five CmuxHomeCore suites pass on a fleet Mac (aws-m4pro-1).

  • backend / test red at bd57de5f544, this branch's own (and on base since #26674): "a part claim left by a vanished Worker lapses" failed under load.

    • The cause: round 6's claim heartbeat ran from the start of a part, and its renewals reset the meter's per-chunk renewal clock. A stalled request whose claim another request had taken then let its bytes through to R2, and the completion was refused.
    • The heartbeat now starts only after the last byte passed the claim checks.
    • The red commit is 4c4f9a1d25f (a deterministic version: [400] where [200, stored] was expected); the fix is aebd8a5d776.
    • At aebd8a5d776: the multipart and direct suites pass 3 runs in a row (36 each); api attachment suites 91 pass; home-core 444; home-server 33; api typecheck clean.

  • backend / test god-file red at aebd8a5d776: home-attachments-multipart.test.ts had reached 1017 lines (limit 1000). It is split at its part-claim seam: helpers go to home-attachments-multipart-support.ts, and the claim and stale-write tests go to home-attachments-multipart-claims.test.ts. No test changed. At f5871469658: 11 files, 91 pass.

  • Review P2 at f5871469658: a part URL signed just before the slot's expiry can finish its PUT after it, and completion then refused the slot. A direct slot's completion and its verification steps are now accepted until the expiry grace, while the key and upload are still kept. New part URLs still stop at the expiry. Red: test/home-attachments-direct.test.ts at 559bc82b98a, which got 403 where 200 stored was expected. Green at a3b0f7cb11d: api 92 pass, home-core 444, home-server 33.

  • Review P2 at a3b0f7cb11d: a non-final part slower than the 15-minute idle lease left the slot expired, and the next part's URL was refused. leaseDirect now takes the slot until the same grace and renews the lease. Red at d9437a16abd: 403 slot_invalid where 200 was expected.

  • Flake seen at 6e2ebc06a85 (run 41119088484): two expiry-abort tests backdate a slot and then drive the wake themselves, but the object's own alarm could run the wake first with the real bucket. Those tests now clear the alarm in the same call that backdates. Green at 2696b153e0c: api 92, with the multipart file run 3 more times (22 each); home-core 444; home-server 33.

Changelog

Fixed: Home attachments whose parts exceed the server's request limit upload straight to storage and are verified before use.

Agent session: agent session

šŸ¤– Generated with Claude Code