#1984 PTY delivery: clear blank-composer residue holding failed-draft latch; dead-letter deliveries stuck behind blocked_on_send

Incident (2026-10-19)

Two broker PTY workers had relay messages stuck for days. desktop-upgrade-sf (sf-mini) held 3 deliveries and desktop-upgrade-finn (finn-mini) held 9, for 42-56h, while each Claude sat idle at an empty prompt.

• /api/status showed about 511-675 resends per delivery, with failed_attempts always 0 and no last_error.
• /api/node-delivery showed surfaced_and_acked: 0 since spawn, and the worker state was blocked_on_send.
• The operator flush (POST /api/spawned/<name>/flush) did nothing (flushed 0, held 0).
• The live snapshot had the cursor at [22,3] on a blank row directly under the prompt row ❯ .

Root cause

1. Prompt not recognised. Claude paints its prompt as ❯ + U+00A0. composer_candidates (broker/delivery_verification.rs is_prompt) only accepted ❯ or ❯ with an ASCII space. As a result current_composer returned None and composer_is_idle was false, even for a truly empty composer. failed_draft_released (pty_worker.rs latch release, ~3113) could therefore never succeed.
2. Blank-row residue. The failed injection left a line break in the composer, so the cursor sat one row below the prompt. That state is not idle either, and nothing would ever clear it.
3. Re-sends reset the failure count. Every 5 minutes (WAIT_DELIVERY_ACK_TIMEOUT) the broker re-sent each held delivery. The worker dropped it as a "duplicate pending delivery" (pty_worker.rs ~1557). Because the handoff succeeded, retry_pending_delivery (runtime/delivery.rs) reset failed_attempts to 0. Nothing ever dead-lettered.

Fix

• is_prompt treats U+00A0 as a space.
• New blank_composer_residue_rows / blank_composer_residue_cleanup. While a failed-draft latch is held and the composer is a bare prompt followed by only blank rows (at most 4, closed by the composer frame rule, nothing at or after the cursor), the worker sends one backspace (0x7f) per blank row. This is bounded to MAX_RESIDUE_CLEANUPS = 3 tries, spaced 2s apart. It never fires when any text exists in the composer, so a human draft is never erased.
• retry_pending_delivery: a successful handoff to a worker whose state is BlockedOnSend no longer resets failed_attempts. It increments the count and sets last_error = "worker blocked_on_send; delivery not injected", so MAX_DELIVERY_RETRIES dead-letters the delivery after about 10 resends. Working workers keep the old behaviour, so a long turn never dead-letters wait-mode messages.
• If residue cleanup is exhausted, the existing failed_draft_latched escalation fires. The latched delivery was already reported failed when it was latched, and the broker budget above then dead-letters the queued ones.

Tests

• blank_composer_residue_holds_the_failed_draft_latch_until_cleared: the real 24-row screen from the sf broker. With the residue it is not idle, residue = 1 row, cleanup = [0x7f]. Once cleared it is idle and the latch releases, which also covers the U+00A0 prompt.
• blank_composer_residue_never_matches_a_composer_holding_text: text on the cursor row, text on the prompt row, and text after the cursor all return None.
• residue_cleanup_is_bounded_and_spaced.
• resend_to_blocked_on_send_worker_counts_against_retry_budget_and_dead_letters and resend_to_working_worker_still_resets_failed_attempts.
• Targeted runs: 63 passed (delivery_verification, residue, retry, latch) and 104 passed / 3 ignored (pty_worker, runtime delivery, composer, injection). cargo fmt is clean.
• clippy -D warnings on toolchain 1.97 fails only on two pre-existing lints in node_control.rs (large_enum_variant, result_large_err), which this PR does not touch.

Open finding (not fixed here)

When the residue was cleared live by hand (POST /api/input/<name> with one \x7f), the latch released, and then every queued delivery (3 on sf, 9 on finn) failed with prompt_unproven. A fresh message afterwards worked.

• The suspected stale-deadline cause is not it. prompt_wait_started is set when the injection is popped (pty_worker.rs ~2327), and prompt_unproven has a single source (pty_worker.rs ~2413) that measures from it.
• The real cause is unconfirmed without the worker logs. Candidates:
  • The manual /api/input write raised the human-ownership / interactive-hold state while queued injections were popped. The window then elapsed while the hold-gated delay arm could not run.
  • A sticky auto_suggestion_visible sent Escape into Claude's empty composer, moving the cursor off the prompt row.
• Worth a follow-up with RUST_LOG=agent_relay::worker::pty=debug.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Agent Relay sessions

• claude session a6d60c2d · opened via gh pr create · last active 2026-10-20