feat(budget-breaker): wake on agent workflow runs, keep the schedule as a backstop

Merged
DO
10d ago
+961-33

Problem

The budget breaker is schedule-only so that only default-branch code ever runs with its write credential. But GitHub delivers almost none of this repository's scheduled ticks: in the first 13 hours after the breaker merged it ran twice (08:35 and 15:03 UTC on 2026-10-11), 6.5 hours apart, against a 10-minute cron. A control that may not run for hours can neither pause the fleet during a quota outage nor, once armed, lift its own pause on time. This blocks arming ("Auto-pause companion", #1565).

Design

The maintainer agreed in principle on 2026-10-11: keep the schedule as a backstop and add a workflow_run trigger.

Trigger. workflow_run, types: [completed], on three workflows in this repository:

WorkflowWhy
"Dev-Lead Agent"Runs the three scripts that post a dev-lead quota marker
"GitHub Agentic Workflows ā€” Cross-Org Dispatcher"Runs two of those scripts directly; no runs in retained history, so no cost today
"Dev-Lead Retry (rate-limited)"Posts no quota marker; listed as one more wake-up that does not depend on agent traffic

pr-review's workflows are deliberately absent: its rate-limited marker is not quota evidence.

Why this keeps the default-branch-only property. GitHub's "Events that trigger workflows" page (read 2026-10-11) says of workflow_run: "This event will only trigger a workflow run if the workflow file exists on the default branch", with GITHUB_SHA the "Last commit on default branch". The same page warns that such a workflow "is able to access secrets and write tokens, even if the previous workflow was not". So the triggering run is treated as untrusted and is never read:

  • The workflow file contains no reference to the event context at all: not in if:, env:, with:, run: or concurrency:.
  • The checkout has no ref: and no repository:. No artifact is downloaded. The script reads the event's name from the runner for its log and nothing else of the event.
  • Permissions, the three credentials and the concurrency group are unchanged.

What an outsider gains. Anyone who can make a listed workflow's run complete here (a comment, a fork pull request, a pull request adding a workflow that carries a listed name) now chooses when the breaker runs. They choose nothing else: quota markers still count only from owners, members and collaborators, and the decision is idempotent.

Cost control: an early exit in the script, not a job-level if:. About half of today's Dev-Lead Agent completions are runs cancelled in their lane. Two reasons there is no conclusion filter:

  • A paused Dev-Lead Agent run has no notice job and ends skipped. Filtering skipped would drop exactly the runs a resume relies on.
  • The concurrency group keeps one pending run and a newer arrival replaces it. GitHub does not document whether a run whose only job is skipped takes that place. If it does, a wake-up that does nothing would replace one that carries news.

Instead, a run woken by workflow_run on a fleet that is not paused exits before the organisation scan when both hold: an earlier run of this workflow was created in the current clock slot (BUDGET_BREAKER_MIN_INTERVAL_MIN, default 2 minutes, 0 switches it off), and this repository shows no quota marker created in that slot. It never applies to a scheduled run, never while the fleet is paused, and never when an input could not be read. The rule is clock slots rather than "the previous run finished less than N minutes ago" because a run that exits early is itself a previous run; under a steady stream of wake-ups that rule would exit early for ever.

Visibility. Every run logs, and puts in its summary, which event woke it and how long ago the workflow last ran to a conclusion.

Risk

  • Run volume. Replaying today's 988 Dev-Lead Agent completions through the concurrency group gives about 47 breaker runs an hour, each a job of roughly ten seconds, most taking the early exit. One runner slot at most is in use at a time.
  • Feedback loop, checked and not found. Dev-Lead Agent also triggers on check_run. If the breaker's own job produced a check_run event that started Dev-Lead Agent, its completion would wake the breaker again. In the two real breaker runs so far (job completed 08:35:45 and 15:05:43 UTC) no check_run-triggered Dev-Lead Agent run followed within a minute. Two data points; see Monitoring.
  • Not observed, from the documentation only: that workflow_run fires when the triggering run ended skipped. It was observed here for cancelled runs (8 of 8 cancelled Auto-rebase runs were followed by their workflow_run consumer within 3 seconds). The skipped case is what wakes a resume; it is added to the arming preconditions in AGENTS.md and can be observed after merge without a pause.
  • Worst cases are improved, not bounded. A trip driven by markers in this repository takes under a minute with a free runner and a few minutes when the job cap is full (one breaker run waited 144 seconds for a runner). Markers only in other repositories, or a quiet repository, still wait for the next wake-up of any kind; today contains a gap of 2 hours 54 minutes between Dev-Lead Agent completions. A resume during a quiet pause can still be hours late. AGENTS.md states this with the measurements.
  • The repository's fork pull request approval setting was not inspected.
  • The breaker stays disarmed. BUDGET_BREAKER_ARMED is unset, and this change does not claim arming is unblocked: the live checks on "Budget breaker: close the gaps found in review before it is armed" (#2251) are still outstanding.

Test plan

  • tests/dev-lead/unit/test_budget_breaker_writer.bats: 97 tests (83 before; 3 new static, 11 new behaviour).
    • Triggers are exactly schedule and workflow_run; every other trigger is rejected, as are types: [requested] and a branches: filter.
    • Every listed workflow name is the name: of a workflow file here, and the list is pinned.
    • No event-context reference anywhere in the workflow; no ref: or repository: on the checkout; no artifact download; the scripts read only the event's name.
    • Early exit: never while paused, never on a scheduled run, fails open on every unreadable input, cannot starve the decision.
  • test_budget_breaker.bats 39 of 39 and tests/test_fleet_pause_coverage.bats 60 of 60, unchanged.
  • Mutation-checked: 31 mutations, each caught by a named test. I re-ran four independently after rebasing (add workflow_dispatch; checkout the triggering SHA; a job if: on the triggering conclusion; early exit moved ahead of the paused check): each turned the expected test red, and the suite returned to 97 of 97.
  • actionlint and shellcheck --severity=warning -x clean.
  • A disarmed dry run of the script against the live API (GET only) printed the trigger and the previous-run line, and took the early exit with a long slot.
  • Not run: anything on GitHub itself. Nothing was dispatched.

Rollback

Revert the squash commit: the workflow goes back to schedule-only. Faster, without a merge: set the repository variable BUDGET_BREAKER_MIN_INTERVAL_MIN to a large value to make nearly every event-woken run exit early (this does not stop the runs). The breaker is disarmed either way, so a rollback changes logging only.

Monitoring

  • Within minutes of merge, "Budget Breaker" runs should appear with event workflow_run. Their summaries show "Triggered by" and "Previous run".
  • Watch the first hour for a loop: breaker runs arriving at a steady interval of well under a minute with no other Dev-Lead Agent activity would mean the breaker is waking itself. Expected rate is about 47 an hour on a busy day, in bursts that follow agent activity.
  • Record on #1565 the first workflow_run breaker run that follows a Dev-Lead Agent run which ended skipped (arming precondition 6).
  • Scheduled runs should continue to appear occasionally; their absence over a day is the delivery problem this change works around, not a regression.

Closes #2259

šŸ¤– Generated with Claude Code