PR #20259 - manaflow-ai/cmux

Summary

Checkpoint capture now recovers local PTY terminals left inside an oversized, unfinished Kitty upload. After bounded replay retries fail, recovery sends normalized CAN through the fenced output path, updates metadata, publishes it to attached parsers, and journals it before capturing a fresh consistency cut.

Completed replayable Kitty chunks are preserved. Recovery fails closed for lost completed prefixes and hosted terminals. The complete capture restarts after recovery so terminal IDs and extension manifests match the new journal cut. Pending journal ingress blocks CAN insertion, and contextual transient errors remain retryable.

This addresses the checkpoint replay portion of #19827; Cloud image promotion is covered separately by #20069.

Testing

Final source: 4035a01db25f1753cc0b5d71ffc7247fe8d9d08f.

python3 scripts/verify-local.py: passed 16 checks; Swift syntax skipped because no Swift files changed. Native compilation and app tests were not run. No localized product copy changed.
./scripts/verify-cmux-tui-hosted.sh --filter checkpoint_recovery_retries_contextualized_ingress_races: run 38497271129 passed on the exact final SHA, including Linux/macOS tests and lint, MSRV, and the macOS arm64 artifact build.
Retry regression: Linux job 115634868283 failed the new assertion on pre-fix commit 246b4619b0e86716203a730593c75d03545e9ce2; the same selector passed after repair. Remaining pre-fix lanes were canceled after preserving that evidence.
Concurrency regressions: run 38495982893 failed both the stale-terminal-list and pending-journal-output assertions on pre-fix commit 0f6bf299f7ccfe4c5ed0bdd0cc21e6fb4ff4e865. The repairs received static re-review; final full-suite execution remains incomplete.
./scripts/verify-cmux-tui-hosted.sh --full: run 38498379099 did not pass. Linux stopped on the pre-existing migration_pairing_code_resumes_old_generation_and_reports_diagnostic timing assertion (1,772 other tests in that binary passed); the core full suite was not reached. macOS test setup was refused by a fleet runner. The wrapper canceled the workflow at its two-hour timeout while Apple artifact work remained; a failed-job rerun ended in startup failure with zero jobs. Windows tests, Linux lint/MSRV, bindings, web, leak checks, and completed cross builds passed in attempt 1.

Review

All four GitHub review threads are resolved. Independent final source review of all five changed files found no verified defects at tree cbf0a6531b29452d147e3c333b0d707201114dc3. Repairs cover replay-prefix preservation, attached-parser and journal synchronization, full consistency-cut refresh, journal ingress ordering, and contextual retries. Review receipts are stored outside source control. Remaining coverage gaps include exit-snapshot recovery, hosted refusal, and the reserved-but-not-yet-active reader case.

GitHub records this exact head as merged by austinywang at 2026-10-14 03:40:46 UTC (merge commit f9d8ad69882ddb4cbd5fc1d8ba2684a11708f676), before full verification completed. This task did not issue a merge command.

Changelog

Fixed: Checkpoint capture recovers from terminal input that cannot fit the bounded replay buffer

Proof

Hosted Rust verification; no graphical app dogfood was performed.

Checklist

Behavior changes have added or updated tests.
User-facing docs are not affected.
Reviewed with a subagent.

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Checkpoint capture now recovers when a terminal is left inside an oversized, unfinished control sequence that can't fit the bounded replay buffer, which previously returned buffer too small forever and blocked checkpoints and journal compaction.

Once normal retries are exhausted, capture detects the unreplayable parser state, sends CAN to return the VT parser to ground, resets the metadata parser, clears only the unreplayable in-flight tracker state, and captures the rendered terminal state while the output fence is held so attached parsers and byte attachments stay synchronized. The synthetic CAN is journaled, so captured journal records include the recovery bytes; capture then re-establishes its consistency cut and recaptures every terminal, refreshing terminal and producer-manifest state, bounded by a maximum number of attempts. Completed upload chunks are preserved so a resumable Kitty upload can still complete later; only an actively overflowed command is aborted. Recovery fails closed once a completed chunk has overflowed the tracker — including when a later control string is itself unreplayable — since CAN cannot recover a lost prefix. Hosted-terminal and live-attach recovery failures are permanent, while recovery races on settled ingress are retried. Addresses the checkpoint replay portion of #19827; the stale Cloud image promotion is covered separately by #20069.

Bug Fixes

Checkpoints can now be created after recovery from replay limits caused by oversized terminal uploads.
When upload content cannot be replayed, the incomplete sequence is skipped while completed portions are preserved.
Terminal output remains usable after recovery, and subsequent terminal metadata can be processed normally.
Checkpoint capture refreshes terminal state and retries after recovery; it fails if recovery does not settle within the allowed attempts.

Summary by CodeRabbit

Bug Fixes
- Checkpoints can now be created after recovery from replay limits caused by oversized terminal uploads.
- When upload content cannot be replayed, the incomplete sequence is skipped while completed portions are preserved.
- Terminal output remains usable after recovery, and subsequent terminal metadata can be processed normally.
- Checkpoint capture refreshes terminal state and retries after recovery; it fails if recovery does not settle within the allowed attempts.