ci(uptime): let the probe past Bot Fight Mode with a per-run IP allow on gauzy.co

Merged
EV
6d ago
+226-4

Why

The External Uptime Monitor has been red on every run since 2026-10-18 11:34Z (last green: run 39122926803 at 11:21Z), and alert #11069 has been open since then. Production was never down. Every probe from a GitHub-hosted (Azure) runner gets HTTP 403 "Just a moment...", a Cloudflare managed challenge, on all five gauzy.co targets.

  • It is not gauzy-specific. ever-co/ever-teams' uptime monitor broke in the same 12 minutes, and ever-works/ever-works' broke about 45 minutes later. The ever.works zone's Security events name the cause: source=botFight, ruleId=bot_fight_mode, managed_challenge on ASN 8075. Its first botFight event is 2026-10-18T11:55:34Z, and the events are retained back to 10-16, so Bot Fight Mode was switched on that day. There is no record of who switched it on.
  • On the Free plan, Bot Fight Mode cannot be skipped by a WAF custom rule (Cloudflare docs: Skip/Bypass/Allow have no effect, because it runs outside the Ruleset Engine). It "will not trigger if an IP Access rule matches the request first". So the original plan (secret header + WAF skip rule) cannot work there.
  • Checks from the workstation return 200 only because the owner's IPs already have user-level IP Access allow rules.

Owner decision (2026-10-20): keep Bot Fight Mode on, and give the probe a temporary per-run IP allow ($0).

What changes (this workflow only)

  1. New step cfallow: looks up zone gauzy.co, logs its plan, and sweeps our own rules older than 20 minutes: IP allow rules whose note starts ever-gauzy-uptime-monitor run , which only this workflow writes. The 20 minutes is longer than the 15-minute job timeout, and the owner's rules (other notes, or none) never match. The sweep does not key on scope.type, which Cloudflare documents only as user/organization. It then creates an IP Access whitelist rule for the runner's own IPv4 egress /32 on zone gauzy.co only, with a note ever-gauzy-uptime-monitor run <id>/<attempt>. If the create's answer is lost, it recovers the rule by this run's exact note so it still gets cleaned up; if even that lookup fails, it marks the run uncertain. This step never fails the run.
  2. Probes and the egress lookup use -4, so requests leave from the allowed address.
  3. New step cfcleanup (if: always()), straight after the probe: deletes the rule, retrying 3 times. It never fails itself, so the alert steps still run.
  4. New last step: fails the run loudly if the rule could not be removed, or if its creation could not be confirmed either way. A later run's sweep deletes a leftover once it is older than 20 minutes (~30 minutes on).
  5. A challenged probe is now reported as HTTP 403, refused by Cloudflare's edge (cf-mitigated: challenge); the origin was not reached instead of a bare HTTP 403, and the alert's reading guide says what that means.

Secret and fork safety

  • CLOUDFLARE_UPTIME_ALLOWLIST_TOKEN is a Cloudflare API token limited to zone gauzy.co, with Zone:Read + Zone:Firewall Services:Edit and nothing else. Only the two steps above receive it.
  • Fork PRs cannot reach it: GitHub gives them no secrets, the job is already skipped for them (if: on the job), and the workflow uses no pull_request_target.
  • With the secret unset, the workflow behaves as before (plus the clearer 403 reason).

Verified

  • YAML parses. bash -n passes on every run: block. Prettier 3.9.10 (repo config) and cspell 10.3.6 (repo config) are clean.
  • Mock Cloudflare API run (fake curl) covering the happy path, a duplicate rule, a lost create answer (rule recovered), a lost answer with the lookup failing too (uncertain), no token, an IPv6 egress, and cleanup success and failure. The stale sweep removed only our own old rule (even organization-scoped); the owner's notes, notes: null, look-alike prefixes and non-IP rules were left alone.
  • Branch dry run without the secret (run 39396564834): every target reported refused by Cloudflare's edge (cf-mitigated: challenge), and the issue was not touched.
  • Still to do: a workflow_dispatch dry run on this branch once the secret exists, then the scheduled runs after merge.

Refs #11069

🤖 Generated with Claude Code

Summary by cubic

Fixes the external uptime monitor, which has been red on every run since 2026-10-18 because Cloudflare's Bot Fight Mode challenges every GitHub-hosted (Azure) runner with HTTP 403 on all gauzy.co targets while real users are served normally. Since Bot Fight Mode can't be skipped by a WAF rule on the Free plan, each run now creates a temporary IP Access allow rule for the runner's own IPv4 egress on the gauzy.co zone, probes over IPv4, and removes the rule right after. A challenged probe is now reported as "refused by Cloudflare's edge (cf-mitigated: challenge)" instead of a bare HTTP 403.

How the allow rule works

  • The rule is zone-scoped to gauzy.co only, noted ever-gauzy-uptime-monitor run <id>/<attempt>, and cleanup runs even when the probe fails.
  • The egress lookup only accepts a real IPv4 reply; an error page moves the loop to the next service.
  • If the create's answer is lost or cut off, the run recovers the rule id by its exact note and IP so cleanup still deletes it; if even the lookup fails, the run is marked uncertain and fails loudly at the end.
  • The stale sweep matches our note prefix only (not scope.type); the owner's rules, null notes, and look-alike prefixes are never touched, and rules older than 20 minutes are deleted. A failed rule listing warns instead of reading as an empty sweep.
  • The run fails loudly when it owns a rule that cleanup did not confirm removed — cleanup failed, or was interrupted before saying so — and when the create was uncertain.
  • The optional CLOUDFLARE_UPTIME_ALLOWLIST_TOKEN secret (gauzy.co only, Zone:Read + Firewall Services:Edit) reaches just the allow and cleanup steps; fork PRs never see it, and without it the workflow behaves as before.