ci(janitor): move jobs off a runner label that stopped starting jobs

TE
5d ago
+3092-117

HOLD: explicit trust-policy approval required for the dedicated repository-scoped route-App token used only for RUNNER_POOLS reads; final exact-head review pending.

Leo: "Bro, I thought we already had logic to automatically route stuff. How can we still have pins like this? I hate pins. Like, just pin the set of the stuff that isn't hilariously slow. Like most of what GitHub has is hilariously slow. We can't be using them."

GitHub binds a runner label when it creates a job. Changing routing variables does not move jobs already queued on an unserved label. The queue janitor now detects an unserved label or a member its pool has retired, then cancels and reruns eligible queued runs only when their workflow's selected runs-on expression reads fromJSON(vars.RUNNER_POOLS).<pool> and that validated pool points elsewhere. Unknown expressions, literal labels, missing pool state, unread listings and opted-out PRs are left alone.

The pass refreshes validated RUNNER_POOLS once and refuses any changed or unread routing before the final run/jobs reads. It accepts only queued/in_progress run states, leaving gated and unknown runs untouched. Each cancel is read once immediately; unfinished obligations resume on the next scheduled sweep without polling. The pass rechecks each run immediately before cancellation. It never moves a started nightly/DEV/promote build, a run with a running job, release/tag/merge-queue runs, or a second attempt. Owed reruns are saved before cancellation in the existing pool snapshot, carried until a confirmed outcome, and share one bounded store with the janitor's earlier recovery passes. A failed read holds all later cancellations. The workflow publishes the snapshot on failure as well as success.

The larger diff keeps one cancellation state machine together: route validation and dead-label detection, one read-failure guard across existing cancellation passes, and shared durable rerun obligations. About half the additions are mocked regression tests. It adds no service or timer and grants no new workflow permissions.

Observed / Not run

Integrated candidate head 1a173e006a88dc495c131bad71caac94442e7231 includes #27129 squash 6ea5ec8f3bd0296fe8222a8ae7bc47539101d1b8. The real pool validator executes with no skips. This remains held pending explicit trust-policy authority and independent final-head review.

  • python3 tests/test_ci_dead_labels.py: 68 passed, zero skips, including the real #27129 validator.
  • python3 tests/test_ci_queue_janitor.py: 274 passed.
  • test_ci_change_areas.test_repository_helpers_route_by_where_they_run: existing assertion red in run 40938219951, focused invocation passes after retaining the conservative all-area route for the owner-run manual test caller. Production routing is unchanged.
  • python3 tests/test_ci_starved_labels.py: 32 passed.
  • python3 tests/test_delivery_workflows_cancel.py: 25 passed.
  • python3 tests/test_ci_cloud_overflow_switch.py: 53 passed.
  • python3 tests/test_ci_workflow_guards_are_wired.py: passed.
  • python3 scripts/ci/validate_test_execution_registry.py --repo-root .: valid, 517 registered tests; pre-existing missing registry entry is warning only.
  • python3 scripts/verify-local.py: all 16 selected runnable checks passed; Swift syntax correctly skipped with no Swift changes.
  • actionlint .github/workflows/ci-queue-janitor.yml .github/workflows/ci-guards.yml and git diff --check: passed.

Test-first adoption regressions prove gated-run races, raw inventory versus runner-selector guards, unfinished cancellation read bounds, and a pool switching back to the dead label. All now pass. Independent review also found capped job listings could hide a running sibling and a corrupt newest archive could fall back to an older empty obligation store. New failing regressions precede the repairs; both pass, and a two-sweep integration test confirms corrupt state holds cancellation and publishes unread markers without emptying obligations. No live janitor apply run, fleet mutation, native build or app launch was performed; no App settings were changed. Independent final-head review and trust-policy authority remain pending.

Trust-policy candidate

The pinned App-token action does not support a variables permission input. For the freshness guard, the trusted janitor script uses the existing overflow-switch pattern: mint from the route App with its installed permissions, explicitly scoped to this repository, then keep that credential in a separate client that permits only GET /repos/{owner}/{repo}/actions/variables/RUNNER_POOLS and accepts no fallback token. The cancellation client and existing token mint callers keep their current permission set. No App setting is changed. The token naturally expires after one hour; the signing helper deletes its temporary key file and no credential is logged. No live mint or apply was executed for verification.

GitHub documents repository scoping and omitted-permission behavior and the Variables read requirement. The mint's exact repository/default/omission payload and the GET-only client boundary are covered by mocked behavioral tests.

  • python3 tests/test_ci_owned_pool_rescue.py Budget: 22 passed, including mint scope/default behavior and temporary signing-file cleanup.

Changelog

none

Agent session: agent session