PR #2281 - petry-projects/.github-private

Problem

"AGENTS_PAUSED stops only dev-lead" (#2249) makes every model-calling workflow skip its work while the fleet is paused. Three monitors then misread such a run, at the moment a maintainer is most likely to be looking at them:

scripts/initiative_canary.sh reports a paused planner run as a healthy canary.
scripts/pr_review_canary.sh labels a paused (skipped) run FAILED.
scripts/fleet_monitor.sh counts a paused run that ended green as a success, so the success rate looks best when the fleet is doing nothing.

Two further defects surfaced while testing the third, both in code that has not executed since 2026-09-20:

The fleet monitor aborted on every run (legacy_attr_file: unbound variable). That was "Fleet Monitor aborts every run" (#2277), fixed by #2278, which merged while this was in review. This branch carried the same three lines so that its end-to-end tests could run; main is merged in and the diff no longer contains them.
Behind that crash, the "Run attribution (by role)" table was built from one record. jq -s 'add // []' over two JSONL files of objects merges the objects key by key, so three runs rendered as ci | 1 | 0 | 1 | 0.

Design

Which signal says "paused". The triage question on the issue was which of three signals is authoritative.

The variable's value when the run started is not in the Actions API afterwards, so it cannot be used.
"Every job skipped" is ambiguous. Live examples from today, with nothing paused: dev-lead run 38148186886 and pr-review-trigger run 38146950206 both ended skipped with all jobs skipped.
So a monitor reads what the run itself recorded: its fleet-paused notice job concluded success. That job runs only when the variable is true.

Four gated workflows have no notice job by design (job-gate-silent: dev-lead, pr-review, ci-failure-analyst, spec-drift), so skipped is all their paused runs record:

pr-review canary: PAUSED requires the dispatched run to end skipped and the canary's own vars.AGENTS_PAUSED to be true. skipped alone stays FAILED, as before.
Fleet monitor: paused runs of those four are left exactly as they were. They were never a success or a failure, and the report says so.

Shape of the change.

New scripts/lib/fleet-pause.sh: pure helpers (no network), shared by the three scripts.
Initiative canary: one extra gh run view --json jobs, only when the run ended success. New status PAUSED: no alert, and not announced as a pass.
Fleet monitor: paused runs are removed before anything is counted and listed in a new "Fleet-paused runs" section. Only green runs of workflows that declare the notice job (or call a registry job-gate reusable) are asked about, 100 runs per GraphQL request. Measured over the last 24 hours here: about 368 candidate runs, so about 9 requests a day. A failed read warns and leaves the runs counted as before.
The workflow file read that the cron parse already made is moved up and reused; it is not a new call.

Item 4 of the issue (the check_run load from notice jobs): not suppressed. Both on: check_run triggers have no name filter, so the runs are created before any if: is evaluated, and both jobs already skip for a notice check run. A name filter would change nothing measurable and would need a dev-lead channel release.

Acceptance criterion 3 (the canary promotion gate in petry-projects/.github): it does not count a paused run. scripts/canary-rollout.sh samples only success and failure conclusions (_tier_sample, lines 1258 to 1270 at b1387cbd), and the only workflows here that standards/canary-rings.json samples are job-gate-silent, whose paused runs end skipped. Caveat: nothing ties the ring registry to the set of workflows with a notice job, so adding a notice-carrying workflow to the registry would make its paused runs count as samples. Nothing was filed there.

Risk

The fleet monitor has not completed a run in three weeks, so the first run after this merges executes a lot of code for the first time in production, not only this change.
The GraphQL read was verified with a maintainer login, not with the monitor's own token. If that token cannot read check runs through GraphQL, the monitor counts as before and warns, whether the request fails or succeeds with checkSuite: null (the second case was silent until the QA advisory on this pull request pointed it out).
No live paused run exists to check against. The paused shape (notice job success, work job skipped, run success) comes from the workflow files; the job names and conclusions were checked on live unpaused runs through both REST and the GraphQL query.
The pr-review canary reads vars.AGENTS_PAUSED seconds before the dispatched run does. A flip in that window mislabels one run; neither outcome alerts.
A run with more than 100 jobs could hide its notice job from the GraphQL read. No workflow here comes close.

Test plan

tests/fleet_monitor_run.bats (new, 15 tests): runs the real fleet_monitor.sh end to end against a stubbed gh. Covers the crash, the attribution table, paused runs removed from totals and rates, a caller of a reusable that carries the notice job, "every job skipped is not paused", a jobs read that succeeds without the jobs, and the batch-size fallback.
tests/test_fleet_paused_run.bats (new, 38 tests): the library, both canaries (pure classifiers and main), and the two workflow conditions.
No existing test file is modified. Existing suites unchanged: test_initiative_canary 16, pr_review_canary 17, fleet_report 93, run_attribution 22, test_fleet_pause_coverage 60.
Mutation-checked: 20 mutations, each turns at least one new test red (late creation of the file, jq -s add restored, no PAUSED branch in either canary, skipped alone read as paused, paused runs never removed, any notice-job conclusion counting, and others). I re-ran three of them independently after rebasing.
shellcheck --severity=warning -x and actionlint clean on every touched file. Both new suites are wired into lint.yml.
Not run locally: the whole repository suite and the Python suites. CI is the authority.

Rollback

Revert the squash commit. The scripts go back to reporting paused runs as before, and the attribution table goes back to one record. The crash fix is separate (#2278) and is not affected.

Review changes

dev-lead pushed one commit to this branch despite the dev-lead:hands-off label (filed as #2294): error handling on one mktemp, a guard on FLEET_PAUSE_BATCH, and a last-line read. Kept after review; the batch guard had no test, so two were added.
From the QA advisory: the monitor now warns when the jobs read answers for fewer runs than it asked about.
Not done here, from the same advisory: saving a real paused run's jobs as a fixture (needs a real pause), and a guard in petry-projects/.github against sampling a notice-carrying workflow (petry-projects/.github#1310 covers the sampler).

Monitoring

Dispatch "Actions Fleet Monitor" by hand right after merge rather than waiting for the daily run.
The next scheduled "Actions Fleet Monitor" run (about 13:00 UTC daily) should end success and post its report, with a "Run attribution (by role)" table that has more than one row.
During the next real pause: the report gains a "Fleet-paused runs" section, the initiative-planner canary reports PAUSED, and neither canary opens an alert issue.

Closes #2266

🤖 Generated with Claude Code