ci(cmux-next): DEV publish and tip run side by side; the newest write wins

Merged
TE
11d ago
+759-45

claude-sr2, relaying fanout-infra: after #25371, cmux-next-dev-daily.yml still has two static concurrency groups, publish (cmux-next-dev-daily) and tip (cmux-next-dev-tip). A static group keeps one pending run and replaces it with the next one. That is the eviction the dev-build rule forbids: "No cancel-in-progress: true and no evicting concurrency group on those workflows". The rule asks instead that a newer tip run alongside, and that a run publish only when it is newer than what the feed already has.

Change. Both groups are gone. Runs go side by side, and every write that could roll DEV back is conditional on R2. The design went through nine Codex xhigh rounds:

  • Round 1 blocked a re-read and retry design: a retry after a write cannot stop a rollback.
  • Round 2 blocked a lease whose expiry used runner clocks, plus three other gaps.
  • Round 3 blocked write-once names that a failed attempt could leave in the way, releases without a commit marker, and unretried 409s.
  • Round 4 blocked empty release notes passing as "no release".
  • Round 5 blocked a feed write that could drop a build published in between, and uploads a losing run left behind.
  • Round 6 blocked a record naming a lost commit, which would deadlock publishing, and write-once uploads failing on 409. It also asked for a rollout that drains old runs.
  • Rounds 7 and 8 blocked deleting uploads after a feed write whose answer was lost, since it may have landed or may still land.

All of these are fixed below.

  • tip (the R2 tip feed):
    • Each attempt's DMG and deltas carry the run id and attempt in their names (cmux-NEXT-DEV-<build>-r<run>-<attempt>.dmg) and upload write-once. Every attempt signs its own DMG, so a feed's signature always matches the bytes behind its URL, and a re-run never collides with an earlier attempt's upload.
    • A run that ends without writing the feed deletes the objects it uploaded, and keeps its exit status. After a feed write that ended without success or a definite conflict, they stay instead: that write may have landed, or may still land.
    • The appcast is a compare-and-swap, and each attempt adds this build's item on top of the feed R2 holds at that moment (tip_feed.py merge), so a build published in between stays. The run reads the feed with its ETag and stops if the feed already has its build or a newer one. It then writes with If-Match (If-None-Match: * when there is no feed yet). On 412 it reads again, up to 5 times.
  • publish (the rolling cmux-next-dev GitHub prerelease; GitHub releases have no conditional write):
    • Lease. The run takes a lease at nightly-next/locks/cmux-next-dev-publish.json (conditional PUTs). Expiry is judged on R2's clock (response Date minus the lease's Last-Modified), so runner clock skew cannot end a live lease. The lease lasts 35 minutes against a 30-minute job timeout. Release is a conditional PUT that marks it released; R2 does not document conditional DELETE.
    • Record. Inside the lease, publish decides from a durable record, nightly-next/locks/cmux-next-dev-published.json, which names the newest commit any run started publishing. The run writes the record (CAS) before the release, so a run that fails halfway still blocks older commits, and a later run for the same commit repairs it. Until the record exists, the release notes stand in for it; a release whose notes name no commit fails publish rather than being overwritten. The run checks it still holds the lease right before writing.
    • Ordering. A newer commit (compare behind) writes; an equal or older one skips. A recorded commit the repository no longer has (GitHub's 404 or 422 for it) is replaced by a source still on the branch, with an annotation; any other error still fails the run. After a rewrite of the source branch (compare diverged), the commit still on the branch wins. Any read or compare error fails the run.
    • Environment. publish uses the existing release environment, which the tip job already uses, for the R2 credentials. It checks out scripts/ci from the default branch.
  • scripts/ci/r2_cas.py (new): get, put --if-match/--if-absent (exit 3 on 412, or on 409 for a concurrent writer), lock, holds, unlock. It reuses upload-r2-object.py's SigV4 signer, which r2-cache.sh and the nightly rollback already use for conditional writes.
  • scripts/ci/tip_feed.py newest FEED prints the feed's newest build, and tip_feed.py merge puts the new item on top of the live feed's items, keeping them verbatim.
  • upload-r2-object.py --write-once treats R2's 409 like a 412: it reads the object, and the digest decides.

Tests: tests/test_cmux_next_dev_tip_feed.py:

  • Workflow structure. No concurrency: in the publisher. Tip archives are write-once and the appcast is CAS-only. publish locks before deciding, and the lease outlives the job.
  • Request headers. put sends If-Match / If-None-Match: * and treats a 412 as a lost race.
  • Lease logic, against a fake R2 whose clock differs from the runner's: take, wait, a runner clock running ahead never ends a live lease, expiry on R2's clock, owner-only release, released means free.
  • Publish decision. A bash harness runs the real publish step with gh and R2 stubbed:
    • A newer commit records then writes.
    • The record outranks stale notes after a half-finished write.
    • A run for the recorded commit repairs the write.
    • A compare error fails.
    • Diverged pairs keep the commit still on the branch.
    • A run that lost the lease never writes.
    • A release whose notes name no commit is never overwritten.
  • Mutations. Removing the record write, or swallowing a compare error, turns the harness red.

These pass locally: that test, test_ci_workflow_concurrency.py, test_ci_self_hosted_guard.sh, and actionlint. fanout-infra extends the no-evicting-group test to dev-daily once this lands.

Rollout. Runs that started before this lands keep the old workflow; every run dispatched afterwards uses the new one. I meant to land only while no cmux-next-dev-daily run was active, but runs now overlap without a gap: from 17:38Z to 18:45Z on 2026-10-26 there was always one running or pending, and main's groups cancelled four of them (40370485774, 40371402817, 40373347803, 40378401600) after sign and verify had passed. So this lands with old runs in flight. The worst case is one old run writing an older commit or tip build over a newer one; the next newer run, minutes later, writes past it (the record and the feed CAS only block older writes).

Risk: medium. This is the DEV publisher's write path. To check after landing: the next dispatched publisher run publishes, cmux-next-dev and the tip feed both name its build, the lease reads released, and the record names its commit. Not exercised before landing: live R2 conditional semantics. Cloudflare documents them for PUT, and this PR no longer depends on conditional DELETE.

Agent session: agent session

🤖 Generated with Claude Code