#24167 feat(monitoring): record Core Features outcomes so long ranges stay cheap

Core Features tiles slow down or time out on 7-day ranges. Each one scans 15–40M samples at that range because pod and instance labels churn (Net change scans 178M), and Prometheus runs on GOMAXPROCS 3 with about 2.7 cores already in use.

This adds recording rules that keep only the labels the tiles use. It changes no dashboard or alert yet.

Measured on prod Prometheus (ro-prod, read-only)

	Samples
Pendant tile at 7 days today	42M
Live session tile at 7 days today	15M
All seven rules, per evaluation	about 150K

The rules add 65 series. There's no remoteWrite on this Prometheus, so no new vendor cost.

Rollout

1. Merge, then the lead installs it. Prometheus doesn't pick this file up automatically:

   kubectl --context gke_based-hardware_us-central1_prod-omi-gke apply -f backend/charts/monitoring/core-features/recording-rules.yaml
   

   To roll back, delete the omi-core-features-recording PrometheusRule. Nothing reads these series yet.

2. Check the recorded series agree with the raw tile queries.

3. A follow-up PR moves the tiles over once the series hold a full 7 days. Moving earlier would make a 7-day tile silently report only the time since install.

Recorded trends smooth short bursts by up to 5 minutes, which is fine for a small trend line.

Verification

• promtool check rules passes.
• Each expression was run as an ad hoc query on prod Prometheus and returned data.
• tests/unit/test_core_features_recording_rules.py passes through backend/test.sh.
• Luna's two findings, the history gap and trend smoothing, are handled by the rollout above and noted in the file. Luna found no other defects.

Requested by David on 2026-10-31.

Failure-Class: none

🤖 Generated with Claude Code