runon: reattach to an accepted job by id after a dropped connection instead of reporting outcome unknown
On 2026-10-30 a ck-motor restart at 01:21Z ended the connection to several runon jobs that kept running on the build server. AFT reported "outcome unknown" for all of them, including two of MC's release gates, and their output was lost. MOTOR confirmed the jobs could have been reattached by job id.
Cause: after a stream error, AFT chose keyed recovery whenever the request had a submission key, even after Accepted had given it a durable job id. Keyed recovery has a shorter path and could run out without ever trying exec.attach.
Fix:
- Accepted jobs: once a job is accepted, a transport loss checkpoints output progress and reattaches by job id. It uses the existing queue-wait allowance, run timeout and grace; nothing new is computed.
- After an AFT restart: restored jobs use the same
attach_remote/ReattachBudgetpath, even when their submission-key attempts are used up. - Daemon restart: attach recovery first reopens the SDK's reconnected exec route. Recovery used to hang dialing another client; the connection file stays as a fallback.
- Before acceptance: keyed recovery and the existing unknown-outcome wording are unchanged.
Tests: reattaching a running job; a job that completes while detached; job-not-found refusal; an unknown attach terminal; repeated disconnects using up the allowance; restart recovery from a persisted cursor; foreground waits; bash_watch; exactly-once terminal delivery; and pre-acceptance keyed recovery. Two existing tests that pinned the old second exec.run now require one exec.run plus a job-id attach.
Verification: 281 remote tests on Linux 12-core plus the Windows compile check.
![View with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png)
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes runon jobs reporting "outcome unknown" after a dropped connection to the build server. Once a job is accepted, AFT now checkpoints output progress and reattaches by job id, returning the real output instead of an unknown result.
- Reattachment uses the existing queue-wait allowance, run timeout, and grace period; restored jobs after an AFT restart use the same path.
- Daemon-restart recovery reopens the SDK's reconnected exec route, with the connection file as a fallback.
- Pre-acceptance keyed recovery and the unknown-outcome wording are unchanged.