fix(tui): a create refused for a full disk leaves no ghost workspace

Merged
LA
5d ago
+581-30

Summary

Fuzz finding restore h1 (STATUS 3083): with the daemon's state directory on a full disk, cmux workspace create --name X answers operation.failed "database or disk is full" and workspace list has no X, but the app shows X (no tabs), also after the app relaunches against the same daemon.

Cause: a terminal-backed workspace.create commits the workspace row and puts the workspace in mux memory (mux/workspace_create.rs) before its terminal and its public projection. When a later write fails, the settlement rolls the staged workspace back with a durable close, which writes to the same full disk. The close failed before it touched memory, and the settlement returned early with ?. AmbiguousLive did not roll back at all. The workspace stayed in mux memory: workspace.list (the store's projection) did not have it, but list-workspaces (mux memory, which the app reads at launch and on every resync) did, until the daemon restarted.

Fix (cmux-tui-core):

  • rollback_interrupted_workspace_creation: when the durable close fails, the workspace still leaves memory (Mux::discard_unpublished_workspace_from_memory). Subscribers get its WorkspaceClosed delta and a TreeChanged resync. The delta keeps the current workspace revision because nothing committed, so the app's delta gate drops it and re-reads the tree on the resync. The close error is logged ({error:#}) and goes to the settlement, so the receipt stays executing: a retry with the same key is blocked as indeterminate, and the recovery at the next start closes the stored row.
  • AmbiguousLive settlement rolls back the workspace the creation staged, and ends its terminal, before it marks the effect indeterminate.
  • settle_unpublished_creation: the unpublished terminal leaves memory and ends even when its tombstone fails, and the workspace rollback runs after a failed terminal close too. A tab, pane or screen in an existing workspace (tab.create_terminal, pane.split, pane.create, workspace.run) still settles as indeterminate, but its terminal ends and its views leave memory first (review R1).
  • close_unpublished_terminal: each view leaves the tree with its close delta (remove_surface_after_registry: TabClosed or PaneClosed), then a TreeChanged resync.
  • settle_failed_creation: when the settlement itself fails (a full disk refuses its rollback or its receipt), the error is logged and the caller gets the creation's own failure (operation.failed, "database or disk is full"), not an internal message (review R2).

PL-daemon owner (cmux-b6, plans/cmux-next/OWNERS.md): this changes the creation settlement in mux/resource_topology/ and adds a memory-only workspace removal in mux/tree_close.rs. Request paths and journal writes do not change.

Regression test

resource_router::partial_create_tests::a_create_refused_for_a_full_disk_leaves_no_workspace_for_the_app. Temp SQLite triggers refuse every registry write with "database or disk is full". The cases are a terminal-backed workspace.create with the disk full after the workspace row, after the terminal reservation and after the terminal is running, and tab.create_terminal and pane.split with the disk full after the terminal is running. Each case checks:

  • (a) the store's projection (workspace.list, public workspace, pane, tab and terminal counts) does not change;
  • (b) the mux state that the list-workspaces tree reads does not change;
  • (c) every add delta has a removal delta that undoes it, and a TreeChanged follows;
  • (e) the caller's error is operation.failed and says the disk is full.
    Two #[cfg(test)] setters reach the existing creation hooks from the router tests.

Command: scripts/ci/cmux-tui-rust-check.sh test a_create_refused_for_a_full_disk_leaves_no_workspace_for_the_app (fleet, Blacksmith Linux)

Observed / Not run

Observed:

  • The red and green jobs above.
  • Clippy (cmux-tui-rust-check.sh clippy, Linux) on the head 1272a5d7a0b7: job 7061df334e5df6a5a2f3cb79, green. Earlier clippy reds on this PR's own lines (48325a7e at eadb73d, 5d6614d6 at 7029f1d) are fixed in 0c522a8 and 1272a5d.
  • The full cmux-tui test suite (cmux-tui-rust-check.sh test, Linux) on eadb73d177ac: job abbade59d0e5dd34cac933a0. cmux-tui-core: 3200 passed, 0 failed. Overall: 14077 passed and 17 failed, all in crates that this PR does not touch, and none in tree_close, workspace_create, creation_settlement or partial_create_tests. These are base findings:
    • cmux-tui cli::cloud_machine_io::tests::* (11 tests), cli::cloud_help::tests::help_lists_exactly_the_offered_cloud_verbs and cli::tests::remote_cli_docs::cloud_help_lists_every_offered_cloud_action: base red from #29928, fix in #30224.
    • acpmux send_attach_family send_message_carries_attachments_to_a_child: base red, the revert of #29670 is in progress.
    • acpmux subsession_trust_inherit a_full_access_chat_lends_its_folder_trust_to_sub_sessions_under_it, acpmux remote_sandbox linux_stack::linux_remote_chain_uses_the_production_sandbox_without_a_switch, cmux-agent-pane-policy typescript the_allowlist_is_what_the_page_sends: base reds (the coordinator saw them on base). Base comparison: job 8977eec831959554b2874ed2 (Actions run 41279042826) ran the full suite on base 09c48ed811e7: 18 failed. 16 of the 17 failures above also fail there. The 17th, acpmux remote_sandbox linux_stack, failed only at the PR run (/tmp/claude-1001: File exists, a runner temp-directory clash); acpmux does not build on cmux-tui-core, and it is a known base red. The base also has 2 failures the PR run does not (acpmux cli::hosts timeout, cmux-tui git_write clean filter).
  • The full suite on 7029f1d16765 (the R1/R2 fix): job eb3d03f0fefc977adc83fe34: 14078 passed, 16 failed. cmux-tui-core 3200 passed, 0 failed. All 16 failures also fail on the base run 8977eec8.

App level (the coordinator ran it): restore fuzz diskfull, seed 4101, with app and acpmux kills, on an exclusive lease from the scheduler, harness 618aac42c55. The fix build is eadb73d177a (job e5a3ba62).

  • Base fb6821e6 (round fgh-h1-ab-base-4101): app_daemon_differ 5 of 5, 5 violations (the ghost "fz-87779-2": [] in the app).
  • Fix (round fgh-h1-ab-fix-4101): 43 of 43 judged, 0 violations, app_daemon_differ 0.

Not run / only read:

  • App's own create path (New Workspace from the UI): read only. A refused create removes its pending row (Packages/macOS/CmuxNext/Sources/CmuxNextApp/WorkspaceSpawn.swift:82 and :133), so it keeps no optimistic row. Its failure text is not localized: ActionWorkFailure("new workspace", error) (AppActions+Workspaces.swift:175) makes "new workspace: " in English (CmuxNextDaemonService/ActionWorkFailure+Daemon.swift:27). That is a separate item.
  • Browser tab creation (tab.create_browser) has no terminal to close: if its publishing commit fails on a full disk, its view can still stay in memory. Not changed and not tested here; a follow-up.
  • A settlement whose read fails (resource_creation_evidence, persisted_creation_settlement, and resource_topology_snapshot in rollback_interrupted_workspace_creation) still returns before the memory rollback. A full disk does not refuse reads, but another I/O error on such a read can still leave a ghost.

Ratchet and base reds at head 6f3b63b2ddf5

  • Crash ratchet: 54b00ebba0c removes the PR's new .unwrap() and .expect( in discard_unpublished_workspace_from_memory. A poisoned state lock takes the function's into_inner path, and a missing close delta is a typed DiscardUnpublishedError that the rollback logs. cmux-tui-core is back to expect 190 and unwrap 1142.
  • Panic ratchet: 6f3b63b2ddf reads the terminal reservation with get instead of a JSON index (indexing_slicing). The value is the same, and a missing key is still None. Panic ratchet db854d90f61138217e4c64ba PASSED at this head (--os linux).
  • Clippy 7e7629cf8810e0f5aa99e02e at this head is red only at cmux-tui-cloud-conversations/src/environments.rs:89 ("boolean expression can be simplified"). That file came with #30357 (7d5f3ae84cc), which this PR does not touch. It is a base red; its revert is #30791.
  • Feat merged at 427778a5ee6 with scripts/merge-main.sh.

Changelog

  • Fixed: A workspace that the daemon refused to create on a full disk no longer stays in the sidebar with no tabs.