Look up the runtime token by type instead of listing every token
Private-pool workers vend each container's runtime BETA9_TOKEN when they claim it. The gateway found that token by listing every non-worker token in the workspace and picking the restricted one in Go. Staging's admin workspace holds 31.8M legacy machine tokens (last created 2026-08-10) and 578k worker tokens, so the lookup tried to load ~32M rows. It never finished within the worker's 5 s claim timeout, the worker retried, and private-pool pods in that workspace never started.
This looks up the newest usable token of the requested type directly, and adds idx_token_workspace_type_created_id on token (workspace_id, token_type, created_at DESC, id DESC) (migration 064, built concurrently with the existing helper that rebuilds an invalid index). RetrieveActiveToken, which pod, endpoint and task queue starts call, also uses the new index.
Measured on the staging database (admin workspace, EXPLAIN ANALYZE):
| Query | Before | After |
|---|---|---|
Runtime token lookup (ListTokens, now RetrieveActiveTokenByType) | over 60 s (timed out) | 0.08 ms |
RetrieveActiveToken (workspace_primary) | 20.2 s, sequential scan | 1.5 ms |
End to end through okteto on staging, with a private runc pool in the admin workspace on the same node:
- With this change: two pods were claimed in the same second their requests were added and served 8.4 s and 1.3 s after deploy. Both received the existing restricted token (matched by hash), and no new token was created.
- Without it: the same pod was received by the worker, never claimed, and served nothing for 92 s until stopped.
Migration notes:
- Staging is already at goose version 64 from this test; the gateway on the previous code starts normally against it.
- The build took about 2.5 minutes on staging (49M rows, 23 GB table, warm cache). The first gateway to start blocks on it under the migration lease; other gateways wait for the lease for up to 4 minutes. Production's token table may be larger.
- Staging's existing
idx_token_workspace_created_id(migration 040) is invalid, so Postgres never uses it. 040 created it without the invalid-index check and an interrupted build was never repaired.
Follow-ups, not in this PR:
pkg/scheduler/pool_local.gocreates aworkertoken in the admin workspace for every k8s worker and never deletes it.- The legacy
machinetokens are still active, and the token list endpoints (pkg/gateway/services/token.go,pkg/api/v1/token.go) still try to return all of them for the admin workspace.
Summary by cubic
Fixes private-pool workers failing to claim container runtime tokens in the staging admin workspace. The gateway found each pod's BETA9_TOKEN by listing every token in the workspace and filtering in Go, but the admin workspace holds 31.8M legacy machine tokens, so the lookup exceeded the worker's 5 s claim timeout and pods never started. It now queries directly for the newest active token of the requested type (RetrieveActiveTokenByType), backed by a new idx_token_workspace_type_created_id index (migration 064). A token is created only when none exists. RetrieveActiveToken also uses the new index.
Measured on staging: the runtime token lookup went from >60 s (timed out) to 0.08 ms, and RetrieveActiveToken from 20.2 s to 1.5 ms. End to end, private runc pool pods were claimed and served within seconds of deploy.
Migration
- Migration 064 builds the index concurrently; the first gateway to start blocks on it under the migration lease (build took ~2.5 minutes on staging).
- Staging's existing
idx_token_workspace_created_id(migration 040) is invalid, so Postgres never uses it.