cloud receiver: renew the policy before its lease ends, accept leases up to 120 s

Merged
TE
2d ago
+137-12

Why

Goal: cmux cloud start-chat --prompt hi on dev reads a reply. On the latest dev capture the machine registers and its Native receiver listens on UDP 4101, but the Mac's link connects and then ends about every 27 s, alternating "connection ended" and "cmux link refused the dial: unreachable" (app log, 12:55:45 to 12:58:08 UTC: ends at +26.4, +26.9, +30.3, +40.5, +18.7 s), until the start gives up with "cloud-chats didn't connect".

The VM's receiver holds its receiving policy on a lease of at most 30 s (expires_at - checked_at, counted from the start of its own read) and renewed it halfway through what was left. A policy read is slow on dev: cmux-platform#645 measured cloud.vm.network_policy at 5.5 to 8.3 s with no peers, and the test team's mesh has about 40 devices now; the receiver's read also mints its token and fetches the keyset. With a read of R seconds, a policy has 30 - R left when it lands, the next read starts after half of that and lands at (30 - R)/2 + R. That is after the lease ended whenever R > 10 s. Each time the lease ends, the receiver drains every stream and removes every WireGuard peer (NativeReceiver::reconcile, expiry timer), so:

  • open daemon connections close (the Mac reads EOF: "connection ended"),
  • dials until the next policy lands find no peer and fail both handshakes ("unreachable"),
  • and this repeats each renewal, a period of (30 + R)/2, which is the ~27 s seen.

Change

  • native_process.rs: the next read starts early enough to land before the lease ends: at least 1.5 times the last read's duration plus 1 s before the end, and never later than halfway through what is left (the old rule, for fast reads). A 250 ms floor keeps one-at-a-time reads from spinning. renewal_delay is a pure function with tests.
  • MAX_POLICY_LEASE_MS = 120_000 (cmux-link native_token): the receiver now accepts leases up to 120 s, in all three checks that held the 30 s cap (NativePolicy::valid, cmux-host's validate, NativeReceiver::reconcile). The backend still sends 30 s; a follow-up in cmux-vm and cmux-platform lengthens it once this image is live, so a read up to about 40 s still renews in time. Revocation does not wait for the lease: every renewal rechecks each stream and peer against the new policy, and a failed read still closes everything at once.
  • The receiver child logs to the journal (stderr) a renewal that landed after the previous lease ended, a failed read, and a read slower than a third of its lease (at most once a minute). Timings only, no ids or keys. Today it logs nothing, so this state was invisible on the machine.

Risk: the VM receiver's authority timing. It accepts longer leases than before, but only from the same authenticated policy read; with today's backend nothing changes except when reads start.

Tests

On the fleet (cmux-ci test, Linux pool, at 92f41aae, the tree of this head): cargo test -p cmux-link (all), cargo test -p cmux-host --lib cloud::receiver, cargo test -p cmux-tui --bin cmux-tui link::native and link::host_, cargo clippy -p cmux-link -p cmux-host -p cmux-tui --no-deps -D warnings: all passed.

  • New: native_process_tests.rs (fast read renews halfway; a 10 s read of a 30 s lease starts after 4 s and lands 6 s early, where halfway landed exactly at the end; a read slower than what is left starts at once).
  • The long-lease refusal tests (vm_receiver_client_read_refuses_long_lease, native_token_receiver_refuses_ambiguous_stale_and_cross_mesh_snapshots) now refuse just past 120 s.

Not run yet: a dev capture on an image with this build (after merge: pin, bake, cmux-vm staging image list, cmux-platform dev image).

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Stops the Cloud VM native receiver from dropping all streams and peers when a slow policy read renews the lease too late. Previously the receiver renewed halfway through what was left of the lease, so on the dev backend a read over about 10 s landed after the 30 s lease ended each cycle, closing open connections and failing dials roughly every 27 s.

  • The next read now starts at least 1.5 times the last read's duration (plus 1 s) before the lease ends, capped at halfway.
  • The receiver accepts leases up to 120 s (MAX_POLICY_LEASE_MS), so the backend can lengthen them; revocation still applies at every renewal.
  • The receiver logs to the journal when a renewal lands after the lease ended, fails, or takes longer than a third of the lease.