fix: prevent client stampede

Merged
LU
3mo ago
+1005-128

Summary by cubic

Prevents client stampedes during sandbox bursts by collapsing stub lookups and making gRPC client creation non-blocking. Hardens lazy image mounting and sandbox pulls for faster, resilient startup; preserves mmap by disabling geesefs direct I/O, uses local workspaces for empty sandboxes, and ensures startup slots are owned by the runtime.

  • Performance

    • Collapse immutable stub loads per node (1s timeout); container clients are reconnectable and non-blocking.
    • Scheduler: wakes on ready pushes, batches GetContainerStatuses, and fast-retries when a worker is pending.
    • Worker/runtime: empty sandboxes mount a local workspace; sandboxes pull remote metadata archives directly and skip embedded cache fill; previously computed required-content is republished on first mount.
    • Cache: spreads immutable OCI layer reads across top-ranked replicas (default 3).
  • Bug Fixes

    • Canceled contexts don’t start stub lookups; sandbox dialing respects cancellation and singleflights safely; startup slots are owned and released by the runtime so cancellations don’t leak capacity.
    • Prevent duplicate container admissions and avoid container-state read/write lock contention.
    • Lazy image mounts wait for readiness with a timeout, clean up on failure, and recover sandbox pulls from the embedded cache if the origin is unavailable.
    • Disable geesefs cache direct I/O to preserve mmap; gateway-vended registry credentials auto-refresh during lazy pulls and agent workers never fall back to node keychains.
    • Cache reporter coalesces recent-stub indexing and only requests reconciliation on flush; shutdown is deterministic with a lease TTL margin.