perf(canary-rollout): cut per-call overhead on the gate's hot path; add opt-in gh timing

Merged
DO
1mo ago
+427-31

Summary

First half of the plan in issue 1259 (Refs below): the Canary Rollout gate-evaluation steps have been ~40% slower since #1238, and 5 of 9 complete scheduled runs now hit the 30-minute timeout-minutes (0 of 10 before). This PR removes a measured, avoidable script cost and adds the instrumentation to find out what else is slow. It does not claim to resolve the slowdown tracked in that issue (see "What this does not prove").

1. Hot-path overhead (measured). _agent_run_json runs once per (agent, repo, window), thousands of times per sweep, almost always in a $(...) subshell. For the common case of a repo with no per-role workflow it cost ~28 ms per call versus ~10 ms before #1238, from two sources:

  • three jq parses of the 57 KB registry per call (run_workflow, the ingress workflow name, ingress_job): ~16 ms;
  • two cache hits, each forking printf | sha256sum | cut and cat: ~14 ms.

Fix: main() preloads those three lookups once, in the parent shell, with a single jq call (subshells inherit the arrays); cache filenames use a fork-free injective percent-encoding instead of sha256 (so A B and A/B still cannot collide; keys too long for a filename fall back to a hash); a tiny cache file (the hot case is a 2-byte []) is read in-shell and anything larger still goes through cat.

Measured with a stubbed gh (200 calls, non-consumer repo, cache primed): main 28.0 ms/call, this branch 4.4 ms/call with the registry preloaded (14.7 ms in the fallback path used when sourced by tests). For a large (670 KB) cached run list the real function is also faster (15.6–18.0 ms vs 19.2–22.6 ms).

2. CANARY_TIMING instrumentation. gh is wrapped in a function (inherited by subshells) that logs start/end per call; _agent_run_json counts its calls. At exit the parent prints one summary per step: wall time, time inside gh vs in the script, calls by kind (gh api grouped by endpoint, top 12), and agent_run_json call count (also appended to $GITHUB_STEP_SUMMARY). It is auto-on only for schedule / workflow_dispatch runs in Actions, off everywhere else (including pull_request test runs), and CANARY_TIMING=1/0 forces it. It never changes an exit status.

What this does not prove

The overhead above is real but modest per call; explaining the full +5 min per step by script overhead alone would need ~20,000 calls per step, and the real call volume and network time are not measured offline. The extra gh run list for the ingress workflow, any tag-lookup retries after #1240, and other costs are not ruled out. That is exactly what the timing summary will show on the next scheduled runs; the issue stays open until the post-merge medians are known.

Risk

Low-medium. Shell logic in the canary gate, but: lookups return exactly what the jq originals return (a test compares them for every agent in the real registry, including a missing agent); the memo is keyed on the registry path and is ignored if CANARY_RINGS changes; nothing is preloaded outside main, so every existing test exercises the original path unless the new CANARY_PRELOAD_REGISTRY=1 test hook is set. Cache filenames change within a sweep's temp dir only (it is per-process and reaped on exit).

Test plan

  • New bats tests: preload parity for every real agent; stale-map guard when CANARY_RINGS changes; a regression guard that a cached non-consumer lookup forks zero jq and makes zero gh calls once preloaded (with a control leg proving the guard is not vacuous); _cache_name safety and injectivity (incl. A B vs A/B, %/_ lookalikes, multibyte, long keys); byte-exact cache hits; _cat_file byte-identical to cat on both sides of its threshold; CANARY_TIMING enable matrix; timing report counts and step summary; timing report with a caller-supplied cache dir; gh api kind grouping.
  • Full suite passes, and again with CANARY_PRELOAD_REGISTRY=1 forcing the memoized path for every existing test. shellcheck --severity=warning clean.
  • No existing test was changed.

Rollback

Revert this PR. No migrations, tags or external state.

Monitoring

On the next scheduled Canary Rollout runs, read the new canary timing block in each step's log / summary: script_time vs gh_time, agent_run_json_calls, and the per-kind table. Compare the two gate steps' wall time to the pre-/post-#1238 medians in the issue (10.4 min per step before, 15.6 after).

Refs #1259, #1175, #1238.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv