fix: drop hard CPU caps; equal PostgREST memory share on ≥2GB instances

Merged
TO
3mo ago
+37-43

Why

Palise Tools (b47a8ea8, ap-southeast, t4g.medium) hit a 45-min window of 10s proxy timeouts and socket resets (feedback 2def74a5). On-box evidence: the postgrest container was capped at 0.3 vCPU / 544MB while the 2-core host sat at load 0.01 — 16,847 throttled cgroup periods, heap high-water mark already at the memory cap, escalating timeout of 10000ms exceeded (0 → 6 → 56/day since Aug 3). Container quotas — not instance class — were the ceiling, so customer resizing cannot fix it.

The legacy 150:50:150 memory split gives postgrest ~14% of RAM on every tier while it runs a RAM-tiered pool (90 connections on a medium), and CPU caps were hard limits that no service could burst past even on an idle host.

What

  • Remove hard cpus: limits from all services. Single-tenant hosts: caps only prevented bursting into idle cores. Under contention the kernel's default equal cpu.weight arbitrates. (Ops follow-up, separate lane: CloudWatch alarm on CPUCreditBalance for burstable instances.)
  • Memory caps stay (host OOM-killer protection) but POSTGREST_BASE becomes 150 on instances ≥1800MB → equal thirds. A 4GB medium: postgrest 544MB → 1268MB (GHC heap cap 1248M). Below ~2GB the 1.0 scale-factor clamp leaves no slack, so nano/micro keep the legacy split.
  • .env cleanup still strips legacy *_CPUS= keys so upgraded instances shed them.

Verification

Simulated the allocator across 460MB–15.7GB totals: nano/micro results byte-identical to current behavior; every tier stays within usable memory:

total RAMpostgrespostgrestinsforgepool
460MB (nano)18461 (unchanged)18415
1900MB (small)62362362345
3835MB (medium)12681268 (was 544)126890
7800MB (large)259025902590150

postgres's reduced cap on medium (1631→1268) is safe: postgresql.conf is conf-driven (shared_buffers=32MB, work_mem=2MB); observed RSS on the incident box is 437MB.

Rollout note: this only affects instances that re-run auto-scale-memory.sh + docker-compose up — existing fleet needs the update pushed (v2.2.9 rollout gap tracked separately).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BpSh9Qb69rJYnKEdMDWPph


Summary by cubic

Remove hard CPU caps and rebalance memory so PostgREST gets an equal share on instances with ≥2GB RAM. This prevents throttling on idle hosts and increases PostgREST headroom (4GB medium: 544MB → 1268MB) to avoid timeouts under load.

  • Bug Fixes

    • Removed cpus: limits from docker-compose.yml; rely on kernel fair scheduling. Memory caps stay.
    • Updated auto-scale-memory.sh to set POSTGREST_BASE=150 when total RAM ≥1800MB; tiny tiers keep the legacy 150:50:150 split.
    • .env cleanup drops legacy *_CPUS keys so upgraded instances shed old caps.
  • Migration

    • Re-run auto-scale-memory.sh and docker-compose up on each instance to apply.
    • Optional: add a CloudWatch alarm on CPUCreditBalance for burstable hosts.

Summary by CodeRabbit

  • Enhancements
    • Simplified resource allocation by focusing on memory limits and reservations.
    • Dynamically adjusts PostgREST’s base memory according to available system RAM.
    • Keeps memory, heap, connection, and pool settings visible in resource configuration output.
    • Prevents memory allocations from exceeding usable system memory.
    • Removed CPU limits from key services, allowing more flexible CPU utilization.

Note on the postgres cap reduction (medium: 1631→1268MB): weighed against the 150-connection ceiling — postgres memory is conf-driven (shared_buffers=32MB, work_mem=2MB per sort/hash), so even 150 connections at several work_mem units each stay well under 1GB; observed RSS on the incident box is 437MB, leaving >2.5× headroom under the new cap.