fix: drop hard CPU caps; equal PostgREST memory share on ≥2GB instances
Why
Palise Tools (b47a8ea8, ap-southeast, t4g.medium) hit a 45-min window of 10s proxy timeouts and socket resets (feedback 2def74a5). On-box evidence: the postgrest container was capped at 0.3 vCPU / 544MB while the 2-core host sat at load 0.01 — 16,847 throttled cgroup periods, heap high-water mark already at the memory cap, escalating timeout of 10000ms exceeded (0 → 6 → 56/day since Aug 3). Container quotas — not instance class — were the ceiling, so customer resizing cannot fix it.
The legacy 150:50:150 memory split gives postgrest ~14% of RAM on every tier while it runs a RAM-tiered pool (90 connections on a medium), and CPU caps were hard limits that no service could burst past even on an idle host.
What
- Remove hard
cpus:limits from all services. Single-tenant hosts: caps only prevented bursting into idle cores. Under contention the kernel's default equalcpu.weightarbitrates. (Ops follow-up, separate lane: CloudWatch alarm onCPUCreditBalancefor burstable instances.) - Memory caps stay (host OOM-killer protection) but
POSTGREST_BASEbecomes 150 on instances ≥1800MB → equal thirds. A 4GB medium: postgrest 544MB → 1268MB (GHC heap cap 1248M). Below ~2GB the 1.0 scale-factor clamp leaves no slack, so nano/micro keep the legacy split. .envcleanup still strips legacy*_CPUS=keys so upgraded instances shed them.
Verification
Simulated the allocator across 460MB–15.7GB totals: nano/micro results byte-identical to current behavior; every tier stays within usable memory:
| total RAM | postgres | postgrest | insforge | pool |
|---|---|---|---|---|
| 460MB (nano) | 184 | 61 (unchanged) | 184 | 15 |
| 1900MB (small) | 623 | 623 | 623 | 45 |
| 3835MB (medium) | 1268 | 1268 (was 544) | 1268 | 90 |
| 7800MB (large) | 2590 | 2590 | 2590 | 150 |
postgres's reduced cap on medium (1631→1268) is safe: postgresql.conf is conf-driven (shared_buffers=32MB, work_mem=2MB); observed RSS on the incident box is 437MB.
Rollout note: this only affects instances that re-run auto-scale-memory.sh + docker-compose up — existing fleet needs the update pushed (v2.2.9 rollout gap tracked separately).
🤖 Generated with Claude Code
https://claude.ai/code/session_01BpSh9Qb69rJYnKEdMDWPph
Summary by cubic
Remove hard CPU caps and rebalance memory so PostgREST gets an equal share on instances with ≥2GB RAM. This prevents throttling on idle hosts and increases PostgREST headroom (4GB medium: 544MB → 1268MB) to avoid timeouts under load.
-
Bug Fixes
- Removed
cpus:limits fromdocker-compose.yml; rely on kernel fair scheduling. Memory caps stay. - Updated
auto-scale-memory.shto setPOSTGREST_BASE=150when total RAM ≥1800MB; tiny tiers keep the legacy 150:50:150 split. .envcleanup drops legacy*_CPUSkeys so upgraded instances shed old caps.
- Removed
-
Migration
- Re-run
auto-scale-memory.shanddocker-compose upon each instance to apply. - Optional: add a CloudWatch alarm on
CPUCreditBalancefor burstable hosts.
- Re-run
Summary by CodeRabbit
- Enhancements
- Simplified resource allocation by focusing on memory limits and reservations.
- Dynamically adjusts PostgREST’s base memory according to available system RAM.
- Keeps memory, heap, connection, and pool settings visible in resource configuration output.
- Prevents memory allocations from exceeding usable system memory.
- Removed CPU limits from key services, allowing more flexible CPU utilization.
Note on the postgres cap reduction (medium: 1631→1268MB): weighed against the 150-connection ceiling — postgres memory is conf-driven (shared_buffers=32MB, work_mem=2MB per sort/hash), so even 150 connections at several work_mem units each stay well under 1GB; observed RSS on the incident box is 437MB, leaving >2.5× headroom under the new cap.