cmux-host: a clock sync is not a resume, so the Cloud receiver stays up

Merged
TE
2d ago
+112-2

Fixes #34328.

Why

On the development backend, a Cloud chat start connects, then each identify times out (identify (no reply within 2.0 seconds), then cmux link refused the dial: unreachable), and the start gives up with "didn't connect". This keeps happening after the mesh ACL is in place: in the last dev capture the ACL went to version 31 with 43 peers at 08:32:37Z, and identify still timed out at 08:33:05Z.

The VM's receiver keeps dying. The provider sets the guest clock on every exec and file operation, and the connect path runs execs on the machine. cmux-host read each clock set (TFD_TIMER_CANCEL_ON_SET) as a resume. It ended remote authority (invalidate_remote_authority), then sent Resumed, and the Cloud role's RoleBridge::resume took a new ticket. Both close the Native receiver child (cmux-tui link __receive-vm, UDP 4101). The child only rebinds after its receiver-policy round trip, 15 to 30 s later. So the device's identify kept landing while nothing listened.

A dev probe measured this on one machine: five execs over 5 minutes. Each exec logged a burst of notify event=resumed, then Native receiving failed: Native receiving bootstrap recipient changed, and ss showed UDP 4101 gone right after each exec. Over the same 5 minutes, CLOCK_REALTIME - CLOCK_BOOTTIME moved by 2 ms (1793415770.934 to .936 s). The provider's clock set is a sync, not a jump.

Change

cmux-host (agent.rs, linux/mod.rs):

  • The Linux platform keeps realtime - boottime and compares it at each clock set. If it moved by at least 2 s (RESUME_CLOCK_JUMP_NS), the wake is ClockSet, as before: remote authority ends, the metadata is read again with the clone signal, and Resumed follows. A snapshot restore (a pause or a fork) stops the boot clock while the machine is stopped, so the set after it moves realtime by the whole stopped time.
  • A smaller move is the new Wake::ClockAdjusted. It still reads the metadata again with the clone signal, so a failed read still fences the identity and a changed instance id still binds again. It no longer ends remote authority or sends Resumed, so the roles and the receiver stay up.

Tests

agent.rs unit tests (cargo test -p cmux-host --lib, on the fleet):

  • a_clock_sync_keeps_remote_authority_and_sends_no_resume: two ClockAdjusted wakes. Neither ends remote authority or sends Resumed, and both still read the metadata again.
  • a_resume_clock_jump_ends_remote_authority_and_sends_resume: one ClockSet ends remote authority once and sends Resumed.
  • only_a_clock_jump_past_the_threshold_is_a_resume: the 2 s boundary, in both directions.

On the base these tests don't compile, because ClockAdjusted doesn't exist there. The behavioural red is the probe above.

Observed / Not run

  • Observed: the dev probe numbers above, and the dev capture log quoted in Why. Fleet unit run on this head: cargo test -p cmux-host --lib 198 passed, 0 failed; cargo clippy -p cmux-host --no-deps clean; cargo fmt --check clean.
  • Not run: a real provider pause and resume with this build. The probe's pause call went to a route that does not hold API-created machines and returned 404. The resume rule is unchanged for any set that moves realtime by 2 s or more against the boot clock. If a resume ever came with less than that, the cost is a skipped Resumed. The announce timer, address-change wakes and the metadata re-read still run.
  • Not run: a new dev capture with this build. That needs this change in the VM image.

Changelog

Fixed: a Cloud machine's link no longer drops for up to 30 s every time the machine runs a command for the app, so a new Cloud chat connects.

Agent session: agent session

šŸ¤– Generated with Claude Code