fix(edge-security): Cloud Armor rejects capture groups; rule counts, quotas, 10 subexpressions

Merged
MI
3d ago
+690-62

Why

Follow-up to #477 (merged), from the live GCP proof that layered-infra-ops is running. The first item is a blocker in the module on main.

  1. Cloud Armor rejects capture groups (blocker). Every client_ip_rules expression used request.path.matches('^(a|b)') and a host pattern with groups. The API refuses them: Invalid regular expression ... Capture Groups are not allowed. So any policy with client_ip_rules fails at apply on main today. The default sign-in rate-limit regex had the same form. The mock-provider tests could not catch this because nothing sends an expression to the API.
  2. A vendor list exceeds Google's default quotas. A new project allows 20 rules with an expression (SECURITY_POLICY_CEVAL_RULES) and 100 rules in all (SECURITY_POLICY_RULES). A host-scoped rule with about 290 vendor ranges planned about 105 and 109, and nothing said so before the apply.
  3. The per-list limit was too tight: 294 ranges, with the vendor list at 292.
  4. A missing API: the Terraform root needs cloudresourcemanager.googleapis.com for data.google_project; without it plan fails with a 403.
  5. Empty-project path: gcloud container clusters create fails in an organization that forbids the default Compute Engine service account for nodes.

What changes

  • Non-capturing groups everywhere. Generated path and host patterns use (?:...), the form the API accepted in the proof. The default rate_limits regex is fixed, and a capture group in a user's path_regex is refused at plan with the reason. A test asserts that no generated expression contains a capture group.
  • max_subexpressions, default 10. Measured: the API accepts 10 subexpressions in one expression and refuses 11 ("Expression count of 11 exceeded maximum of 10"). A rule now carries 9 client ranges, or 8 with hosts, where it carried 4 or 3. The 290-range list needs about 45 advanced rules, not about 105.
  • rule_counts output (advanced, total) on the module and in terraform/examples/gcp output.edge, so terraform plan shows what the policy needs against the quotas.
  • Docs: "Quotas: check before you apply" in the module README and the same warning in docs/gcp-install/3-EDGE-HARDENING.md (see the per-policy limit below).
  • Priority bands of 1000 (client rules 10000+, rate limits 20000+, allowlist 30000+): a list can hold 2,500 ranges and 999 rules. Relative order is unchanged.
  • Guide: cloudresourcemanager.googleapis.com in step 2's gcloud services enable; a node service account for organizations that forbid the default one.

Effect on existing policies

Expressions and priorities change, so a policy applied from #477 has its client, rate-limit and allowlist rules replaced on the next apply. No such policy can exist with client_ip_rules (it could not be created), and the only known policy is the proof stack's.

Tests

terraform test (mock provider), 34 runs in the module and 12 in the example root: the exact host and path patterns the API accepted; no capture group in any generated expression; a capture group in path_regex refused, (?:...) and escaped parentheses accepted; rule_counts equals the number of rule resources; 8 ranges per host-scoped rule at 10 subexpressions; the Cloudflare default needs 5 advanced rules and 8 in all. Local: terraform fmt/validate/test, tflint, terraform-docs and checkov pass. No chart change, so no version bump.

The per-policy limit (added after the first live apply)

After the project quota was raised to 100, the full policy still failed: Quota 'SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY' exceeded. Limit: 20. That quota is fixed; a request to raise it is refused. One policy holds at most 20 rules with an expression. So:

  • A precondition (max_advanced_rules, default 20) now refuses a plan that needs more, before anything is created, with the count and the options.
  • With the secret header, two WAF rules and sign-in and staff rules, about 13 rules remain for a vendor list: roughly 100 ranges. A 293-range list cannot fit in this policy at any setting. The docs say to merge ranges where the wider block is acceptable, or enforce that vendor's list at the fronting proxy. Address groups (Cloud Armor Enterprise) are not supported by this module.
  • The docs also quote the project-quota error, say that it is granted automatically in seconds when requested (measured on an existing project), and warn that a failed apply leaves a partial policy that must not be attached.

A caller with the secret could skip the proxy (added after the policy was attached live)

With the policy attached, the proof found that a request admitted by a client rule never reaches the allowlist of connecting addresses, so the shared-secret header was the only guard. A caller holding the secret, connecting straight to the load balancer, was admitted and chose its own client address through CF-Connecting-IP: as a staff address, and as a vendor on a vendor path.

  • New rule at priority 150, ahead of the WAF and of every rule that admits: deny a connection that comes from none of connecting_asns (Cloudflare: 13335) or connecting_address_groups (an address group of the proxy's ranges, Cloud Armor Enterprise). The proxy's 15 ranges do not fit one negated expression.
  • client_ip_rules are refused at plan without one of them, unless secret_header_only = true accepts the header as the sole guard.
  • Path prefixes now end at a path segment (/mcp no longer matches /mcpfoo).
  • Guide: the backend-services list showed the policy about 4 minutes before every backend enforced it, so the check now sends a request per host. trustedProxies: 3 behind Cloudflare is now measured.
  • terraform/examples/gcp: labels on the private Cloud SQL DNS zone.

Proven live vs not

Proven on the proof stack with the policy attached (at commit b39e76ab): the non-capturing patterns, 10 subexpressions, a clean apply at exactly 20 advanced rules, both quota errors, the secret header rule, vendor path scoping (admitted on the listed paths, denied on 12 others), the sign-in limit (60 then 429), throttles counting per CF-Connecting-IP, WAF preview logging, hosted MCP end to end.

Proven on a second live run (at commit 60afa9ae, connecting_asns = [13335], Standard tier): rule 150 denied 31 of 31 direct requests that carried the secret, from three vantage points, whatever CF-Connecting-IP claimed; 0 of 668 requests through Cloudflare were decided by it (all arrived from AS13335; two Cloudflare locations sampled, so not every location is proven). The path-segment boundary: /mcp, /mcp/x admitted; /mcpfoo, /mcp.json, /MCP and encoded-slash tricks denied. A plan after the apply shows no changes.

Not proven: the address-group form of the guard (needs Cloud Armor Enterprise). Not run, because the proof's Cloudflare token cannot edit rules or certificates: the Cloudflare transform rule that adds the secret header (the client supplied it in the tests), Full (strict) with an Origin CA certificate, and a connection held open past 100 seconds.

Known limits, documented: a new rule takes up to about 4 minutes to be enforced on every backend; the connecting check costs one of the 20 rules; another customer of the same proxy who holds the secret still arrives from the same network.

šŸ¤– Generated with Claude Code