ci: treat a reused app-host receipt PID as a stale receipt, not a cleanup failure
App-host cleanup no longer fails a finished shard when the dead app host's PID has already gone to another process. Shard 5/7 of full-suite run 36680987910 passed all 711 tests, then failed its cleanup step with FAIL: app-host receipt does not match the PID executable vnode. That was one of the red jobs tracked in #15488.
Receipts stay on disk after the app host exits, and a busy fleet Mac reuses PIDs quickly. In that job, app-host PIDs went 85306 → 97393 → 5596 → … → 8949 between 07:12 and 07:16, so the PID space wrapped twice. By cleanup time the receipt's PID belonged to an unrelated executable. cmux_verify_app_host_receipt treated that as a hard mismatch, even though elsewhere the same file already treats a PID running another executable as exited.
The receipt descriptor is O_CLOEXEC, so no exec keeps it. A live PID that runs another executable and demonstrably does not hold the receipt is therefore a new process.
- What changes. When the receipt's PID runs another executable, cleanup asks
lsofwhether the PID holds the receipt.lsofexits 1 both for "nothing found" and for errors, so that exit alone proves nothing. The receipt counts as absent only when a second, unfilteredlsoflists the PID with at least one open file, and none of them is the receipt file on any descriptor. The receipt then verifies as stale (2), which authorizes and signals nothing. A PID that exits between the two queries is also stale. The stale-receipt check gets the same change. - What still fails. Cleanup still fails when a PID holds the receipt under another executable (on any descriptor), when the listing shows no open files, and on any other
lsoferror. It now printslsof's diagnostic in those cases. The live-target scan still fails on any app host without a verified receipt.
Verification
tests/test_ci_app_host_processes.sh drives the check through a fake lsof. After the fix (f8cfb8629b3) it prints PASS: app-host processes require matching receipts and executable vnodes.
| Case | Expected | Result before the fix |
|---|---|---|
reused-pid: /bin/sleep has the PID and lists its files without the receipt | stale, nothing signaled | e0133e6bea3: FAIL: a receipt whose PID was reused by another executable failed cleanup |
uninspectable-pid: the receipt query exits 2 | cleanup fails | the first version of this fix treated it as stale |
unlisted-pid: the receipt query and the listing both exit 1 | cleanup fails | the second version treated it as stale |
fileless-pid: the listing names the PID but no files | cleanup fails | 8e4bf66c5df: a listing without open files was treated as a stale receipt |
moved-receipt-fd: the receipt is open on fd 10, not fd 9 | cleanup fails | 8e4bf66c5df: a receipt held on another descriptor was treated as stale |
exiting-reused-pid: the PID exits between the two queries | stale | 8e4bf66c5df: failed cleanup: ... does not match the PID executable vnode |
Coverage. CI runs this script only in the Linux app-host-process guard, against the fake lsof. The macos job is skipped for this PR, since the suite chooser selects no app-host run for these paths, so no app-host shard has executed the change. I ran the new code against the real /usr/sbin/lsof on macOS 26.4:
- a process holding the receipt on fd 9 returns 0 and is not treated as reused;
- a process holding it on fd 10 returns 1 and is not treated as reused;
/bin/sleepwithout the receipt returns 3 and is skipped as stale;- an exited PID returns 2.
The reviewer got the same results independently on the previous head. The next main full suite is the first run on fleet Macs.
Changelog
- none
🤖 Generated with Claude Code
Summary by CodeRabbit
- Bug Fixes
- Improved handling of stale app-host process records. When a process identifier has been reused, the stale record is no longer treated as authorization to signal the live process.
- Process verification now distinguishes exited or reused processes from cases where inspection fails or a receipt cannot be confirmed, avoiding unsafe actions when verification is incomplete.