fix: prevent client stampede
Merged
LU
3mo ago+1005+1005+0123456789012345678901234567890123456789-128-128-012345678901234567890123456789
Summary by cubic
Prevents client stampedes during sandbox bursts by collapsing stub lookups and making gRPC client creation non-blocking. Hardens lazy image mounting and sandbox pulls for faster, resilient startup; preserves mmap by disabling geesefs direct I/O, uses local workspaces for empty sandboxes, and ensures startup slots are owned by the runtime.
-
Performance
- Collapse immutable stub loads per node (1s timeout); container clients are reconnectable and non-blocking.
- Scheduler: wakes on ready pushes, batches
GetContainerStatuses, and fast-retries when a worker is pending. - Worker/runtime: empty sandboxes mount a local workspace; sandboxes pull remote metadata archives directly and skip embedded cache fill; previously computed required-content is republished on first mount.
- Cache: spreads immutable OCI layer reads across top-ranked replicas (default 3).
-
Bug Fixes
- Canceled contexts don’t start stub lookups; sandbox dialing respects cancellation and singleflights safely; startup slots are owned and released by the runtime so cancellations don’t leak capacity.
- Prevent duplicate container admissions and avoid container-state read/write lock contention.
- Lazy image mounts wait for readiness with a timeout, clean up on failure, and recover sandbox pulls from the embedded cache if the origin is unavailable.
- Disable
geesefscache direct I/O to preserve mmap; gateway-vended registry credentials auto-refresh during lazy pulls and agent workers never fall back to node keychains. - Cache reporter coalesces recent-stub indexing and only requests reconciliation on flush; shutdown is deterministic with a lease TTL margin.