Speedup vDSO CNTVCT and amortized urandom

Merged
JS
5mo ago
+5386-296

vDSO clock_gettime drops from 1256 ns SVC trap to 2.5ns via CNTVCT-based fast path (493x speedup, 20x under the sub-50 ns design target). The trampoline emits a 28-instruction A64 sequence that reads CNTVCT_EL0, LDAR-acquires the vvar initialized flag, and interpolates wall clock from the anchor as delta * 125 / 3 (Apple Silicon CNTFRQ = 24 MHz), falling back to SVC on first call or CNTVCT regression. The first SVC seeds the vvar via a three-state CAS (0 -> 2 -> 1) so concurrent first calls cannot tear the anchor fields. The seed is gated on ELR_EL1 matching the trampoline's svc_fallback PC so an unrelated raw clock_gettime syscall cannot poison the anchor from arbitrary X9.

/dev/urandom 1-byte reads drop from 5688 ns uncached to 2054 ns (2.77x) via a new per-fd entropy cache: an arc4random_buf-refilled 4 KiB buffer per FD_URANDOM slot. The cache is zeroed on close via a type-to-cleanup registry that also closes pre-existing dup and fork-state race windows for every synthetic fd type.

eventfd dup shares state across aliases per the Linux contract (refcounted slot plus eventfd_owner[FD_TABLE_SIZE] table). The dup path holds fd_lock and sfd_lock together for the bind commit so racing close cannot leak the refcount; the source identity is pinned via snapshotted host fd so a racing close-and-rebind of the source cannot bind to the wrong slot. tests/test-eventfd-dup pins the shared-state contract.

fork_ipc_send_fd_table filters eventfd, signalfd, timerfd, inotify, netlink, pidfd, and epoll out of the SCM_RIGHTS payload. macOS rejects kqueue fds across SCM_RIGHTS and per-class side-table state is not transferable, so a clean drop is the only honest contract. tests/test-fork-synthetic-fd pins it.

Startup decomposition: ELFUSE_STARTUP_TRACE=1 emits per-step wall time for VM bring-up (17 steps on test-hello, dominated by hv_vcpu_create and guest_init at roughly 0.9 ms each). Zero overhead when unset.

Summary by cubic

Adds a CNTVCT-based vDSO fast path for clock_gettime and EL1 shim fast paths for identity syscalls and 1‑byte /dev/urandom reads. Delivers major speedups (clock_gettime ~500x; urandom 1‑byte ~2.7x; identity ~47 ns) and hardens isolation with an EL1‑only shim_data block and stricter infra guards.

  • New Features

    • vDSO: versioned ELF with GNU symbol versions (LINUX_2.6.39), CNTVCT fast path for __kernel_clock_gettime, single SVC seeds REALTIME+MONOTONIC, publishes VDSO_OFF_SIGRET, resolves with glibc/musl.
    • EL1 shim: EL1‑only shim_data (identity cache, attention mask, urandom ring + readable‑FD bitmap). Serves getpid..getegid in EL1 (gettid via CONTEXTIDR); 1‑byte urandom in EL1 with data‑abort recovery; TPIDR_EL1 set on exec/fork; attention recomputed in the vCPU loop and tied to signals/itimers and cred publishes.
    • /dev/urandom: keep 4 KiB per‑fd host cache with per‑fd locks; bitmap mirrors readable urandom FDs; readv also refills the ring; cache/bitmap reset on close/dup/fork.
    • Infra/boot: shim_data mapped MEM_PERM_RW_EL1_ONLY and shown as PROT_NONE in /proc/self/maps; ELF loader rejects PT_LOAD/PHDR writes into infra; mremap/madvise reject infra ranges; startup tracing via ELFUSE_STARTUP_TRACE=1.

  • Bug Fixes

    • eventfd dup shares counter/readiness across aliases.
    • Fork: drop eventfd, signalfd, timerfd, inotify, netlink, pidfd, and epoll from SCM_RIGHTS; child recreates them; no host‑fd leaks.
    • Central FD cleanup registry applied atomically for synthetic types; urandom bitmap publication serialized.
    • getrandom validates flags and returns EINVAL for unknown bits.