fix(canary-rollout): a capped per-role run list no longer loses the front of a window

Merged
DO
23d ago
+287-25

Summary

The legacy per-role reader (gh run list -L 1000, date-bounded to 15 days by #1282) accepted a list that hit 1000 even when its oldest run was newer than the window start. On the busiest workflows the newest 1000 runs cover less than a day, so:

  • a failure since the candidate cut dropped out of the gate's view once 1000 newer runs existed, and
  • the 14-day baseline zero-filled every day the list never reached, which sized the sample at its clamp minimum.

This is question 5 of the solution-architect advisory on #1244.

Evidence (live, read-only, 2026-10-11)

CheckResult
Newest 1000 runs of .github-private "Dev-Lead Agent"reach back to 01:09Z the same day
Failures in a 5-day window, newest-1000 list alone0
Failures in the same window with this change33
robust_sample_target 250 3 15 3 437 0 0 0 0 0 0 0 0 0 0 0 0 0 (zero-filled)3
Same day alone (unseen days dropped)15

What changes

A per-role list at its cap that does not reach back before the window start (run_list_covers, a new pure function) is no longer trusted as complete.

  • Gating read. The window's failure-class runs are read with a server-side filter (--status failure, --status startup_failure, same date bound) and merged in. The executed count from the general list becomes a lower bound, which can only delay a promotion. If a failure-class list is itself truncated, the member is UNRESOLVED.
  • Baseline read. The oldest listed day is written to the truncation flag _baseline_daily already arms, so unseen days are unknown, not zero. This is the treatment a capped ingress read already gets.
  • No window start. The list is returned as is.
  • The two ingress call sites now use run_list_covers too. Their behavior is unchanged.
  • The UNRESOLVED blocker note and ::error:: line no longer assert that the member is an ADR-0007 collapsed repo, since a legacy member can now be UNRESOLVED.

Behavior changes to review

  1. Busy agents at next->ring0 are sized from the days the list shows. For a workflow with hundreds of executed runs a day the target moves from 3 to 15. At that volume 15 runs accumulate well inside the 4-hour dwell floor.
  2. A truncated list with nothing countable is the non-zero floor (target 3), not "no caller". Before, a busy workflow whose newest 1000 runs were all skipped had its sample waived. This inherits the ingress rule; item 16 on #1244 (minimum versus maximum for an unknown baseline) is still open and is not decided here.
  3. Pairs with failures older than the newest 1000 runs will now show them. Expect some pairs that read clean today to go BLOCKED on the next sweep. That is the point of the change.
  4. One pinned assertion is flipped on purpose. The #1282 test "a capped bounded list triggers no refetch" asserted exactly one run list call. It now asserts three (the general list plus two status-filtered reads) and still asserts no unbounded refetch. The test is renamed to say so.

Cost: two extra list calls per member and workflow, only when the general list is truncated for the window. They share the 15-day bound, so the #1271 cache serves every window in the tier.

Test plan

  • bats tests/canary_rollout.bats: 10 new tests, 496 of 498 pass locally. The two failures (grep -P in the #1054 and #1023 tests) are a macOS BSD grep limitation and fail the same way on unmodified main.
  • shellcheck --severity=warning -x scripts/canary-rollout.sh scripts/lib/canary-rollout.sh clean.
  • Read-only run of _agent_run_json against live .github-private history (table above).
  • CI on Linux.

Refs #1244

🤖 Generated with Claude Code

Problem

See the linked issue.

Risk

Low — changes automation shell logic under scripts/, covered by shellcheck (--severity=warning) and the bats suite.

Test plan

Tests added/updated: tests/canary_rollout.bats. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

This PR's Lint (shellcheck) and bats checks show pass/fail; watch subsequent dev-lead / pr-review runs for behavioral regressions.