acpmux: machine join and consent locks unlock on drop (flock flakes)

Merged
TE
8d ago
+37-8

Flake fix for reds seen on base: cli::machine_consent::tests::tags_share_enrollment_and_live_grants_and_revocation failed at machine_consent_tests.rs:18 with "consent update in progress" in 7 of 60 full cargo test -p acpmux --lib runs at base 13531e30. cli::cloud_join::tests::concurrent_join_is_refused_before_a_second_redeem failed once in those runs at cloud_join.rs:557. Both come from #27216. Fixing base reds; never blocked.

The cause

Both locks are a nonblocking flock on a file that is taken, dropped by closing its descriptor, then taken again. flock belongs to the open file description, and a process spawned at that moment shares the description from its fork until its exec. Other lib tests spawn processes all the time, and so does the daemon (agents, tools, login shells). Closing our descriptor therefore left the lock held for that moment, and the next take failed as "consent update in progress" or "another machine join is in progress". #27407 fixed the same race in a test of host_approvals.

The daemon hits it too. record_auth_result takes the join lock on each authenticated cloud response, and a consent change takes the write lock, both while the daemon spawns. A change from the Settings switch could fail with "in progress" when nothing else was changing consent.

The fix

  • join_lock and the consent write lock now return HeldLock, which unlocks explicitly on drop. The upload record (attach/upload.rs) and the Codex auth broker already do this. The unlock acts on the open file description, so it releases the lock whatever copies remain.
  • Each test now holds a copy of the lock's descriptor across the drop, as a spawned child does. The race is now taken on every run. The assertions are unchanged: a second take still fails while the lock is held and succeeds after the drop. Consent changes from both tags still apply.

Evidence (Linux, cmux-ci test, 32 vCPU)

runtreeresult
3 x 20 full cargo test -p acpmux --libbase 13531e30tags_share_… failed 2, 4 and 1 of 20; concurrent_join_… failed 1 of 20 (1, 2, 3)
10x a 2000-step grant-then-revoke loop, 8 threads spawning beside it (instrumented, not committed)base 13531e30524-606 of 2000 steps failed with "consent update in progress", in every run (run)
5x the two tests with a held copy, production unchangedbase 13531e30 + this PR's testsboth failed 5/5 (run)
10x the same loop and both test files, with the spawning threadsthis head0 of 2000 steps failed, 10/10 (run)
30x full cargo test -p acpmux --libthis headmachine_consent and cloud_join passed 30/30 (run)
cargo clippy -p acpmux --all-targets -- -D warnings, cargo fmt --checkthis headclean (same run)

The other reds in those runs fail the same way at base, and each has its own PR:

  • protocol::tests::protocol_ir_is_current: deterministic, the spec has been stale since #27216.
  • wireguard::tests::loopback_throughput_and_shutdown (#27434).
  • session_scope::tests::a_session_start_never_waits_for_another_to_set_the_slice (ETXTBSY).

Risk: low. A drop-time unlock on two short-lived locks. HeldLock keeps the file open for exactly as long as the File did.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes intermittent flock failures in acpmux where machine join and consent locks appeared held after being dropped.

  • Closing the lock's descriptor left the lock held whenever a spawned process (agent, tool, login shell) shared the open file description from its fork until its exec, so the next nonblocking take failed with "consent update in progress" or "another machine join is in progress".
  • Locks now unlock explicitly on drop via a new HeldLock type, releasing the lock for the open file description regardless of copies.
  • Tests hold a copy of the lock's descriptor across the drop to reproduce the race on every run.