fix: bound jobstore copy connections per process

Merged
GI
36m ago
+299-34

What changed and why

Concurrent cold privacy purges created a new two-session pool per invocation, multiplying connections outside the namespace cache. The environment-selected jobstore factory now shares two normal sessions and one reserved erasure session across namespaces and cached/cold calls in each process. Normal saturation cannot consume deletion capacity; failures still raise into durable retry, and retargeting requires a process restart.

This is a per-process bound. Fleet admission, operating budgets and production rollout remain separate; no workflow, deployment, IAM, migration or production setting changes are included.

Product invariants affected

none

How it was verified

  • Ten focused unit files, each in a separate pytest process: 297 passed. The new counting-connector regression covers normal saturation, simultaneous cached/cold purges, connection/configuration/SQL/partial failures, cancellation and retry.
  • Disposable PostgreSQL16.16 plus loopback Firestore emulator: purge11, index8 and production-copy29 passed (48 total), with required backends and no skips. Both services cleaned up. All data synthetic; no production calls.
  • Image source-closure check passes all19 registered images. A direct connection-module import avoids traversing the package facade's unrelated exports. Repository make preflight at head 68120ee8846fd9644faae90a02012c9cb94c194c passes41 checks. No gate bypassed.

Tests

backend/tests/unit/test_jobstore_copy_connections.py drives the real pool and factory. backend/tests/integration/test_jobstore_purge_postgres.py now exercises erasure through a separate reserved connection, including mode-off cold calls. Existing index/cohort and current-source retirement suites pass.

UI evidence

UI-Evidence: none -- internal backend resource ownership only.

Failure class (fixes)

Failure-Class: new

Failure-class transition narrative

The violated contract is a shared resource ceiling multiplied by per-call pool construction. The new FC-per-call-pools-multiply-shared-capacity definition names the shared process-owned factory and concurrent counting-connector test as the canonical prevention. This patch provides the first source fix and guard; no previous merged PR is asserted as evidence.