ci(janitor): no-op CI runs stop evicting the waiting sweep; a long sweep still publishes

TE
7d ago
+41-6

Summary

The queue janitor writes the macos-pool-load snapshot that PR placement reads. Placement wants that snapshot to be under 15 minutes old, but on 2026-10-30 it was older than that for 54% of the day (607 of 1,133 minutes, measured from generated_at). The worst gap was 50 minutes, 17:28Z to 18:18Z. This PR fixes two causes on the writer side.

  1. No-op runs pushed real sweeps out of the queue. The workflow-level concurrency group allows one running run and one waiting run, and each new arrival cancels the one that is waiting. A CI run of a pull request into feat-cmux-next starts no sweep (the job's if), but it still joined the group. Those runs made up 238 of the janitor's last 400 runs. Each one cancelled the real sweep waiting there, crons often within seconds of being queued, and then skipped. After 17:57Z every new arrival was one of these no-op runs, so nothing swept until the 18:14 cron. Such a run now gets a group of its own (ci-queue-janitor-noop-<run_id>). Its condition is the same expression as the job's if, so every run that does sweep still shares ci-queue-janitor, and sweeps still run one at a time.
  2. A timed-out sweep published nothing. The upload step runs only when the job is not cancelled, and a job timeout counts as cancelled. Since #28402 a sweep lists the whole queue, which took up to 13 minutes. Two sweeps in a row, started at 17:36Z and 17:47Z, hit the 10-minute timeout, so neither published. The job timeout goes from 10 to 15 minutes.

This changes only the writer. Placement's handling of a stale snapshot is #28249's.

Not addressed here, for a follow-up: GitHub's cron drift (median 11 min between */10 ticks today, up to 25.5), and the fact that the 300/600 s skip is timed from the last upload while age is measured from generated_at.

Testing

  • python3 -m unittest tests.test_ci_queue_janitor: on the base, the two new WorkflowShapeTests cases fail (test_a_run_that_starts_no_sweep_never_evicts_a_waiting_sweep, test_a_long_sweep_still_publishes_its_snapshot; red commit b229081a04a). On the head, all 269 tests pass.
  • python3 -m unittest tests.test_ci_workflow_run_api_budget tests.test_ci_fork_runner_routing tests.test_nightly_delivery_concurrency: 70 tests pass.
  • actionlint .github/workflows/ci-queue-janitor.yml reports nothing.
  • python3 scripts/verify-local.py --affected origin/main: 16 of 16 checks pass.
  • Review: codex gpt-6-astra (high) through router-review returned SHIP at d31e4754fe0ec2875fef49eb2450229e9544554d.

Observed / Not run

  • Observed: the gap analysis above, from today's janitor run list and macos-pool-load artifacts.
  • Not run: a live sweep on this branch. Schedule and workflow_run events run main's copy, so the effect shows only after merge.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the queue janitor's macos-pool-load snapshot going stale (up to 50 minutes old) by fixing two causes: no-op CI runs cancelled the waiting sweep, and timed-out sweeps published nothing.

Bug Fixes

  • Runs whose job's if starts no sweep (CI runs of PRs into feat-cmux-next) now get a per-run concurrency group, so they can no longer cancel the real sweep waiting in ci-queue-janitor.
  • The sweep job timeout increases from 10 to 15 minutes, because a timed-out job skips the upload step and sweeps listing the whole queue took up to 13 minutes.

Summary by CodeRabbit

  • Chores
    • Updated automated maintenance workflow handling so concurrent pull request runs targeting feat-cmux-next use separate concurrency groups, while other events retain the shared group.
    • Increased the maintenance sweep timeout from 10 to 15 minutes.
    • Added checks for the workflow’s concurrency behavior and minimum timeout. No user-facing product changes.