perf(canary-rollout): date-bound the Agent Ingress run list (the dominant gate cost)
Summary
Second half of the plan in issue 1259 (Refs below), driven by the first scheduled run that carried the CANARY_TIMING output from #1260 (run 37581060882):
| gate step | wall | time inside gh | script time | gh run list calls | their total | avg / max |
|---|---|---|---|---|---|---|
| promote-all | 745 s | 740 s (99%) | 6 s | 58 | 607 s | 10.5 s / 31 s |
| sync-issues | 769 s | 763 s (99%) | 6 s | 58 | 593 s | 10.2 s / 30.5 s |
Every other gh call kind is cheap (tag lookups 0.38 s avg, matching-refs 0.43 s, ...). gh run list is ~80% of each gate step.
Cause (from the code; the timing kind could not yet say which workflow). The Agent Ingress run list of each collapsed repo is read with -L 5000 (CANARY_INGRESS_RUN_LIMIT), so gh pages through up to ~50 pages per repo, although every window the gate samples starts at most ~2 weeks back (the 14-day baseline is the widest by default). That list was introduced in #1238.
Fix. The ingress list is now fetched with --created ">=<date>":
_ingress_horizon <since>snaps the window start to a small set of day tiers (CANARY_INGRESS_HORIZON_TIERS, default8 15: the candidate/correctness windows and the 14-day baseline). Older or empty windows, and aSOAK_WINDOW_DAYSbeyond the largest tier, stay unbounded, exactly as before._repo_wf_runs_cachedtakes an optional 5th arg (the date). It is part of the cache key, so bounded and unbounded lists never serve one another, and windows in the same tier share one fetch.- GitHub caps any workflow-run listing that uses
createdat 1,000 results (found in review)._ingress_runs_fetchtherefore trusts a bounded list that reaches 1,000 runs only if its oldest run is before the window start (everything newer is then in it); otherwise it reads the unbounded list, i.e. exactly the previous behavior. For such very busy repos there is no saving, and no loss of history. - The timing report now labels run lists as
run list [<workflow>] L=<cap> [bounded], so the next scheduled run shows directly whether the ingress lists were the slow ones and whether the bound removed the cost.
What this does not change (semantics)
- Which runs count: the list is only trimmed of runs older than every window that uses it; each caller still cuts at its own
since. The horizon date is the start of the daytdays back, witht >=the window age in days rounded up, so it is never later than the window start (tested for 0-15 days back). - The fail-closed cap guard is untouched: a bounded list that hits
CANARY_INGRESS_RUN_LIMITbefore the window start is judged exactly like an unbounded one (tested), and the 1,000-result ceiling above is handled by the fallback. - Per-role (legacy) workflow lookups are not bounded.
What this does not prove
I could not measure ingress run volumes offline (the repos' Actions data is not readable from this session), so the size of the saving is an estimate: if the lists are dominated by runs older than 14 days, the 10 s calls should fall to ~1 s, i.e. up to ~9 min per gate step. If the volume is mostly inside 14 days, or a repo exceeds 1,000 runs in its window, the saving will be smaller for it. The relabelled timing output from the next scheduled run settles it, and issue 1259 stays open until the post-fix medians are known.
Risk
Low-medium: it changes what the promotion gate fetches. Mitigations: the bound is conservative (rounded up to a tier, unbounded beyond), a possibly-truncated bounded list is never used, the existing ingress tests are unchanged and green, and new ones pin the new behavior. Rollback is a revert.
Test plan
- New bats tests: tier mapping (including the empty-tiers disable) and the "never later than the window start" invariant; bounded for a recent window, unbounded for an old/empty one; per-tier cache sharing (3 fetches for 4 windows, none repeated); cap guard on a bounded list; the 1,000-result fallback and the "reaches back, trusted" case; run-list timing labels. One earlier #1260 timing test updated for the new, more specific
run listlabel. - Full suite 448/448, and again with
CANARY_PRELOAD_REGISTRY=1.shellcheck --severity=warningclean.
Rollback
Revert this PR, or set CANARY_INGRESS_HORIZON_TIERS="" (empty) on the workflow step to disable the bound without a revert.
Monitoring
On the next scheduled Canary Rollout runs read the canary timing block: run list [Agent Ingress] L=5000 bounded calls/avg/max, and the two gate steps' wall time (10.4 min before #1238, 15.6 after, 12.4/12.8 in the first sample with #1260).
🤖 Generated with Claude Code