Retry runnerless nightly runs after three minutes

Merged
TE
6d ago
+5-5

Runnerless NEXT nightly runs currently wait 20 minutes before the existing bounded cancel-and-rerun rescue. A hosted-light control can therefore hold a nightly well past the requested three-minute no-start window. Lower the runnerless no-start threshold to three minutes while preserving all existing safety checks: only runs with no started work qualify, pending gates stay untouched, owned labels remain with owned-pool recovery, and cancellation still requires an existing rebuild path plus a fresh second read.

Fixes #30237. Review receipt: https://github.com/manaflow-ai/cmux/pull/30173#issuecomment-6553703766

The merged rescue path remains the reroute mechanism: canceling a never-started tip/older nightly or promotion run causes its existing path to rebuild it, re-evaluating the runner expression on the next attempt. GitHub does not support changing one queued job's runner label in place, and started work is never canceled by this path. This change does not add hosted labels, pins, a new cancellation path, or whole-run cancellation for started work.

Validation: python3 -m unittest discover -s tests -p test_nightly_next_stuck_run.py — 24 passed. Regression cases cover 1/2 minutes remaining queued, cancellation at 3/10 minutes, started and in-progress work protection, skipped/timed-out jobs, gates, owned labels, fresh rechecks, and bounded settle behavior. git diff --check passes.

PerformanceBeforeAfterDeltaEvidence
Runnerless no-start rescue threshold20 minutes3 minutes-17 minutes24 focused rescue tests
Started nightly cancellationProtectedProtectedUnchangedStarted/in-progress regression tests
Hosted/Blacksmith runner selectionExisting expressionExisting expressionUnchangedNo workflow runner selector changes

Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Lowers the runnerless no-start rescue threshold from 20 minutes to 3 minutes so a hosted-light control can't hold a nightly past the requested no-start window. Fixes #30237.

  • Only runs with no started work qualify; gated runs, owned labels, and in-progress jobs remain protected.
  • Cancellation still reroutes through an existing rebuild path and requires a fresh second read before acting.

Summary by CodeRabbit

  • Chores
    • Nightly Next CI runs now cancel eligible stuck jobs after 3 minutes instead of 20 minutes, allowing stalled runs to be cleared sooner. Jobs within the 3-minute window remain unaffected, and existing eligibility checks are unchanged.
    • Updated the related wait-time guidance and automated checks to reflect the new threshold.