fuzz(restore): diskfull and --visible chaos modes, host/close evidence (fuzz row h)

Merged
LA
5d ago
+925-19

Summary

Restore and chaos fuzz harness (fuzz catalog row h), the next step after #29376. Harness only: it adds no CI job, step or gate, and it changes no product code.

  • diskfull chaos mode: the slot daemon's state dir is moved onto a bounded HFS+ image (--disk-image-mb, default 512) mounted at the same path, and the image is filled to --disk-margin-mb (default 2). The image file is in the slot TMPDIR, never in the run output.
    • The setup refuses to run if the host would keep less than 30 GB free.
    • Every --disk-recover-everyth iteration runs with the filler removed, and that iteration must run again.
    • A restore that fails on the full disk frees the space and checks that the build recovers by itself. If it does not, the harness presses Restart cmux-tui (debug.daemon restart, the button's path) and records whether the app offered Restart.
    • Refused creates count as degraded, with each error and whether it is typed.
    • On every exit (finally, SIGTERM, SIGHUP), the harness detaches the image, puts the original state dir back and removes the image file. Each run first detaches a stale fzdisk image that an earlier run left in the slot HOME.
    • An iteration starts only if its waits can end inside the lease.
  • --visible: from each kill to the settle, the harness samples debug.daemon and debug.surfaces every 250 ms and judges them with the rule of restore-journey.py RecoverySampler: no blank pane; each window shows content or the connecting view; terminals are connected, restoring or reconnecting; both calls answer. It also records the states seen.
  • Evidence: terminal hosts vs the daemon's listed terminals (hosts.jsonl), each close with its time, and listed minus shown minus closed-in-the-last-30 s (this was the live proof of the host-leak fix #29476). Also: the app/daemon topology difference after refused work, and exact daemon env through KERN_PROCARGS2.

Changelog

none

Testing

  • python3 scripts/fuzz/restore/test_restore_fuzz.py: all tests pass. The new DiskFull, VisibleState, TypedErrors, Hosts and Closes tests were written first and were red before the code.
  • python3 scripts/cmux-next/prepush-gate.py: no new violations.

Observed / Not run

  • Observed, through scheduler lease rounds on build fb6821e6 (feat d1f0e63691a0):
    • diskfull seeds 4001, 4002, 4004, 4005 and 4006 found h1 (STATUS 3083) and h2 (STATUS 3149).
    • Host-leak A/B: listed-minus-shown-minus-closed30s max 248 on the base, 6 on the fix.
    • cpu + --visible, seed 4003: 13 of 14 pass. The 1 fail was a 1.02 s answer gap while the sampler itself ran under CPU starvation; I did not file it.
  • Harness faults found on the fleet and fixed here: a copy back of the image held a lease for about 4 min, and one iteration ran past the lease.
  • Not run: the last commit (lease-bound iteration start) on a mini.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Extends the restore fuzz harness (fuzz catalog row h) with a diskfull chaos mode, --visible sampling, and host/close evidence. Harness only: no CI job, step, or gate, and no product code changes.

diskfull chaos mode

  • Moves the daemon's state dir onto a bounded HFS+ image (--disk-image-mb, default 512) filled to --disk-margin-mb free (default 2); the image lives in the slot TMPDIR, not the run output, and the run refuses to start when the host would keep under 30 GB free.
  • Every --disk-recover-everyth iteration runs with the filler removed and must run again; refused creates count as degraded with the error and whether it's typed, and a refused iteration that also saw a stalled answer or any other problem is a failure (all modes).
  • A restore failing on the full disk frees the space and checks the build recovers by itself; when it does not, the harness presses Restart (debug.daemon restart) and records whether Restart was offered; recovery after an app kill keeps the daemon and acpmux, so only one app instance runs on the state dir, and short leases record "not checked" or "Restart not tried".
  • Every exit (SIGTERM, SIGHUP, SIGINT) detaches the image, restores the original state dir, and removes the image file and stale fzdisk images an earlier run left mounted (the build is stopped before detaching); each step runs even if another failed.
  • A run whose verdict list shows no kill (only refusals or not‑run iterations) is NOVERDICT; kill counts never crash on records without a target.
  • An iteration starts only if its waits can end inside the lease.

--visible and evidence

  • --visible samples debug.daemon and debug.surfaces every 250 ms from kill to settle and judges them with the restore-journey RecoverySampler rule: no blank pane, each window shows content or the connecting view, terminals are connected/restoring/reconnecting, and both calls answer (for an app kill, from the new app's first answer).
  • hosts.jsonl records terminal hosts vs the daemon's listed terminals, each close with its time, and listed minus shown minus closed-in-the-last-30 s (the host-leak proof of #29476); after refused work it also records the app/daemon topology difference and the exact daemon environment through KERN_PROCARGS2.