cmux NEXT: one queue that brings other machines' daemons to the app's build

Merged
TE
10d ago
+906-5

Part 4 of the rebuild of #23199 and #24070 (adopted by hb-acpmux). Builds on #25386 (merged), and is rebased onto feat-cmux-next.

The queue that brings other machines' daemons to the app's build. Nothing offers it a machine yet: acpmux ssh peers and Cloud machines are the next PRs.

Leo: "BRO CANT YOU JUST LIKE SUSPEND SHIT AFTER A TOOL CALL/TURN THEN JUST GO AND AUTOUPDATE OUR TEAMMATES NEED ACPMUX UPDATING TOO..."

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

Leo-ask: ask-48bbaec1d4
Leo-ask: ask-061e3574a6

Change

DaemonUpdatePolicy (from #24070 at bd4892c8), one per app (shared), fed by offer(_:):

  • Trigger: a machine on another commit than the app's bundled build. A dirty bundled build pushes nothing. An urgent machine (a Cloud machine the app refused) goes even with no known build, and goes first.
  • Never a move back (new since #24070): a machine on a build of a later day, or on a build with local changes (someone's own), is left alone. #24232 made Linux hosts follow feat-cmux-next by themselves, so moving one back to the app's build would only restart it twice.
  • No wait for idle. A machine is updated mid-turn as soon as the policy reaches it, and the restarted daemon resumes what it interrupted, as Leo's words above ask (#24232 made the daemon persist and resume). updateBusyAfter defaults to zero; a non-zero value (idle asks first, mid-turn at the cap) stays an option that nothing sets. #24070's last version did the same.
  • A machine that can't update now (unreachable, another update running) is asked again with backoff: 2 minutes, doubling, at most 30 minutes apart. After 6 hours it is left for the next launch.
  • One machine at a time across every source. An id already queued or running is not queued twice.
  • One attempt per machine and build (DaemonUpdateLedger). A final outcome (updated, current, rolled back, failed) is kept across launches, so a rollback is not retried before the next build.
  • Each step goes to DaemonUpdateBoard. A recorded failure asks for one notice.
  • The toggle agent.updatePeers is read before every step, and turning it off empties the queue.

Timers and their stop conditions

  • The drain loop. It ends when the queue is empty, which happens when every machine is done, left for the next launch, or the toggle went off.
  • The backoff nap (clock.sleep, at most 30 minutes). It ends at its deadline or when a new offer cancels it. Its machines stop after 6 hours of not being able to update.

Tests

DaemonUpdatePolicyTests (fake machines, an auto-advancing test clock):

  • an old machine is updated; a machine on the build, of unknown build or nogit is left alone; a dirty bundled build pushes nothing;
  • aTargetOnANewerOrItsOwnBuildIsLeftAlone (red first: 03733e83 is #24070's policy, which updates them; CI at that commit's earlier copy f56f2c5b showed this test failing);
  • aBusyTargetIsUpdatedMidTurnOnTheFirstAsk (red first: 59e2fbad has these tests on the 10-minute default, and this test fails there; the next commit sets the default to zero);
  • withACapANeverIdleTargetIsUpdatedWhileBusyAtTheCap: with the optional cap, idle asks at 0, 2 and 6 minutes, then mid-turn at 10;
  • aTargetBusyAlsoWhileBusyIsLeftAfterSixHours: 2, 4, 8, 16, then 11 Ɨ 30 minutes, all mid-turn; unrecorded; it leaves the board;
  • one at a time;
  • a retried machine that reached the build meanwhile leaves the board;
  • a failure is recorded and asks for one notice, also after a relaunch; a rollback is retried only on the next build; no binary is not recorded; a machine handled elsewhere leaves no state;
  • an urgent machine goes first and doesn't wait behind a busy one's backoff;
  • offers are deduped while running; the toggle off runs nothing; a machine offered while the toggle-off is reported still updates.

Observed / Not run

  • CI at c7b8f0a0 (run 40301784433): DaemonUpdatePolicyTests and DaemonUpdateTargetTests passed, among them aBusyTargetIsUpdatedMidTurnOnTheFirstAsk, aFailureWhoseLedgerCannotBeWrittenIsNotRetried and aBundledVersionThatNeverAnswersEndsAtItsBound. The daemon tests passed.
  • The swift test job is red the way it is on other PRs today: the test process is killed (exit 143), and the failing tests (AppKit commit boundary, sidebar rows, panel warm-up, browser tab mirror, ActionRunCreated) are in code this PR does not touch and fail on other PRs' runs too.
  • Not run: no real daemon or peer was updated.

No capture: no visible UI.

Agent session: big-red
šŸ¤– Generated with Claude Code