ci: nightly failover and janitor exemptions cover delivery runs only

Merged
TE
7d ago
+157-21

Summary

dep-fleet's ruling (#27521) says: never cancel a started nightly, DEV or promote build. It protects delivery runs, meaning runs that publish, sign, roll back or file things. It never meant that a pull request's test run of a post-publish check can't fail over. Three exemptions read it by workflow path or name instead, so the pull request runs of nightly-next-canary.yml and nightly-next-launch-smoke.yml (both trigger on pull_request for edits to themselves) were covered too:

  1. owned_pool_rescue.never_fails_over() exempted every path in NEVER_FAILOVER_PATHS and any file name containing "nightly", so the open-head failover never moved a pull request run of the canary or launch smoke off a dead label.
  2. queue_janitor.PROTECTED_WORKFLOW matched "nightly" in the run name or path, so the sweep left every canary and smoke run alone, pull request runs included.
  3. queue_janitor.reserves_pool() (#28364) matched delivery workflows by path only, so queued pull request jobs of the canary and smoke counted as reserved_queued. The 14:16Z snapshot showed reserved_queued: 8 on glaeda-side-std-xcode-26.6.

One shared predicate fixes all three: delivery_workflows.delivers(path, event, *names). It is true for a delivery workflow's run (is_delivery_run(), unchanged) from any event except a pull request's (pull_request, pull_request_target, merge_group). An unread event still counts as a delivery event, so a run whose event we can't read keeps its protection.

  • Rescue: Target now records the run's event. never_fails_over(path, event) is true only for a delivery run: a run of a NEVER_FAILOVER_PATHS workflow or any delivery workflow, from an event that can deliver. Runs of nightly.yml, DEV daily, release.yml, sign, train, rollback, verdict and feed-watch keep their exemption. Most of those never have a pull request run anyway, and the failover already skips non-PR targets.
  • Janitor: protected_reason() drops the "nightly" match only for a pull request run of a delivery workflow. Release, publish, notarization, TestFlight and App Store names still protect it. orphan_protected_reason() and reserves_pool() use delivers().

Nothing changes for scheduled, dispatched or pushed runs, nightly-branch runs included: they stay protected and reserved. A started nightly delivery build is still never cancelled.

The diff is small and touches never_fails_over() and the Target constructors only. #28249 (feat-cmux-next) does not touch those lines.

Testing

Red on base (0368f4e1, tests only on top of 7ca718f7), green on head:

  • python3 tests/test_delivery_workflows_cancel.py, DeliveryRunTests. Base: 32 errors (delivers missing). Head: OK (29 tests).
  • python3 tests/test_ci_owned_pool_rescue.py, three tests. A pull request run of the canary or smoke is not excluded. A PR run of the canary or smoke on a dead label is cancelled and re-run. Every delivery run (schedule, dispatch, push, workflow_run, unread event) is still excluded. Base: 2 failures ('cancel' not found in ['pull', 'run']) and 2 errors. Head: OK (403 tests).
  • python3 tests/test_ci_queue_janitor.py, DeliveryRunScopeTests. A PR run of the canary or smoke is unprotected, classified stale-pr when its PR is closed, kept at its open head, and reserves 0. Scheduled, main and nightly-next dispatch runs and a nightly-next push of nightly.yml stay protected and reserve 1. Base: 2 failures ('release/nightly/TestFlight/App Store workflow' is not None). Head: OK (253 tests).

Observed / Not run

  • Observed: the three files above, plus every other tests/*.py that imports queue_janitor, owned_pool_rescue or delivery_workflows (health report, owned pool sources, late placement, pool pickers, starved labels, nightly delivery concurrency, nightly-next feed watch and cancelled tip, run_e2e, app-host product reuse), all pass on head.
  • Not run: a live sweep. The janitor and rescue run from main's copy on their next schedule.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Limits the nightly failover exemption and janitor protection to delivery runs, so a pull request's test run of the nightly canary or launch smoke workflow is treated like any ordinary PR run.

Adds delivery_workflows.delivers(), true for a delivery workflow's run from any event except a pull request's (pull_request, pull_request_target, merge_group). An unread event still counts as a delivery one, so protection stays when the event can't be read. The rescue's never_fails_over(), the janitor's protected_reason()/orphan_protected_reason(), and reserves_pool() all use it.

  • Before, exemptions matched "nightly" by workflow path or name, covering PR runs of nightly-next-canary.yml and nightly-next-launch-smoke.yml; those runs publish, sign, roll back and file nothing, so dep-fleet's ruling (#27521) never protected them.
  • Scheduled, dispatched, pushed and unread-event delivery runs stay protected and reserved; started nightly builds remain uncancelled.

Summary by CodeRabbit

  • Bug Fixes
    • Pull request, pull request target, and merge-group runs of delivery workflows are now handled as test runs rather than delivery runs. These runs can follow the usual stale pull request and failover behavior.
    • Scheduled, manually started, pushed, and workflow-triggered delivery runs retain their protection and queue reservations.
    • Delivery-run handling now accounts for the triggering event and run details, avoiding incorrect protection based only on a workflow’s name.