runon: recovery allows the runner's real queue wait before declaring a job lost

Merged
AF
2d ago
+491-54

After a disconnect or a daemon restart, AFT re-attaches to its runon jobs and gives up only once the job's limits have passed. It assumed every job waits at most 300 s in the build server's queue. When no queue limit is sent (AFT sends none), ck-motor allows 300 s only for Boat-eligible jobs (8 vCPUs or fewer, no network); every other job may wait up to min(1800 s, its run timeout). So AFT could give up on a queued 16-vCPU job and report its outcome as unknown while the runner still held it.

Recovery and submission-key expiry now share one helper, recovery_queue_wait_allowance = max(300 s, run timeout). That bound covers either runner policy without copying ck-motor's routing rules. The 300 s re-attach grace, the hinted-retry cap after a proven refusal, and the 24 h key retention are unchanged. AFT still sends no queue_wait_limit_s.

The give-up message changes one clause, from "ck-motor's queue wait limit of 5m" to "a queue wait allowance of 30m" (the number varies by job), because the old text would now be wrong. The rest of the message is unchanged.

Tests: large_queued_runon_recovers_after_the_small_queue_deadline failed before the fix ("queued job abandoned", FateUnknown, after 41 injected minutes for a 16-vCPU job with an 1800 s runtime), and also pins recovery of the runner's real output without a second exec.run. Further tests cover pre-start refusals (runner_full, network_unsupported, queue_wait_exceeded, capability and connection failures) ending the foreground wait and bash_watch at once, which already worked on main.

Checks: the worker's build-server runs were cut off (a compile timeout, then a lost result channel), so the filtered tests and the Windows compile haven't run anywhere yet. CI is the gate.


Summary by cubic

Fixes AFT giving up on queued runon jobs after recovery by bounding queue wait with the runner's real admission policy instead of a fixed 300 s assumption.

  • Recovery re-attach and submission-key expiry now share a conservative queue allowance of max(300 s, the run timeout), covering both Boat-eligible and hosted-only queue waits.
  • The give-up message now reports "a queue wait allowance" of the job-specific value instead of "ck-motor's queue wait limit of 5m".
  • Pre-start refusals and pre-submission failures still end the foreground and worker waits promptly; post-refusal retry caps, the 300 s re-attach grace, and 24 h key retention are unchanged.
  • Adds a benchmark harness slice covering the recovery behavior; no search or indexing code changed.