Scale batch GPUs from admissible queued work

Merged
GI
1d ago
+1034-69

Parked allowance holds periodically re-enter ready for policy evaluation while keeping their original enqueue time. The existing age-only HPA could therefore request the GPU ceiling for one old held row. Use a fresh indexed count of admitted first attempts and allowed retries, and keep raw age as an SLO diagnostic.

New admission records compute_decision.outcome=unattempted. Existing worker versions already overwrite that with held/allowed or remove it on an unevaluated refusal, so mixed versions cannot leave a separate capacity eligibility bit true. The count also requires ready, merge-pending and currently available work. Unmarked legacy debt remains durable and drainable by warm workers; paid compute still requires the existing per-lease funding/privacy authorization.

Production values retain four fixed workers with autoscaling and balanced pools off. The prepared cutover renders two pool-pinned deployments/HPAs, each min1/max12 (total2/24), with separate ownership/scrapes and one shared metric scope. The adapter deduplicates workers before summing the three priorities, requires all three fresh classes and never fabricates zero. AverageValue divides the shared demand across both pools. Deployment reporting rejects a missing, unready, zero-sized or wrong-image Spot companion. The query driver and generated prod/dev composite indexes are included.

Only the dedicated worker's supervised sampler runs capacity aggregates. Failed capacity queries retain the prior values and freshness without failing recovery/projection requests. The pinned Firestore SDK decodes a genuine integer-zero aggregate as 0.0; that exact result is accepted, while missing, fractional and malformed results remain unavailable. The real loopback gauntlet caught this boundary; focused tests now exercise the installed SDK decoder. Synthetic gauntlet processes retain failure tracebacks in private process logs, separately from sanitized evidence.

Validation: targeted backend files run through backend/test.sh with explicit file lists under omi-heavy/nice19. The18-file pass found one stale adapter-test assumption about the production default; the final five-file rerun covers that correction, capacity, chart, metrics and characterization. The other business, runtime, readiness and seven Firestore guard files passed; query-shape coverage includes511 cases. The full batch family of the hermetic Sync Cloud Tasks gauntlet passed, including recovery, process drain/kill, privacy, burst and two workers. A prior burst admission503 did not recur and remains an unresolved test finding; it is not claimed fixed. Canonical formatting, diff hygiene and all38 make preflight checks passed.

Characterization goldens intentionally add only final_job.compute_decision.outcome=unattempted:75 additions across11 files. Structural comparison confirms no other golden changes; paid-call counts, outcomes and transitions are unchanged. First review required preserving fixed4 until qualification; that fix is included. Final delta review and new-head CI remain pending. The initial CI development-index prebuild passed add-only validation then failed WIF authentication; the release owner identifies that as the existing PR-event identity condition issue, with no IAM change proposed here.

Activation hold: queued-unit target16 is provisional slot arithmetic, not qualified throughput, and is unused by the production default. Before enabling the scaler, benchmark useful completions and queue latency on the exact worker and pin the measured target. The benchmark owner is bramp3, review deadline2026-10-28 09Z; the release owner approves the measured values and op1 applies them. No production configuration, admission cohort, GPU allocation, account data or paid provider was changed by this PR work. Verify the new indexes READY, both worker roles/images and the live external count before the approved cutover. Capture the current release first; rollback restores its pinned image/configuration and four Ready workers with normal SIGTERM drain. The fixed4 fallback remains in force until qualification.

Regional/global GPU quota was independently checked read-only: global12000 limit,22 allocated GPU quota units across43 regions; us-central1 L4OD160/16used and Spot32/1used. This is quota headroom, not stock availability or a billing/savings claim. No new workflows or IAM changes.

Cutover requires both the CPU admission producer that records unattempted and the GPU sampler image. A worker-only image roll can export a fresh zero count while older producers still create unmarked work; it remains a fixed4 instrumentation roll. Verify the actual serving producer revision, new-unit marker/count and new worker provenance before enabling either HPA. The observer's artifact pins and source qualification must follow the approved CPU/worker rollout; source merge is not installed/runtime acceptance.

Failure-Class: none