Optimize anonymous mmap and page-fault paths
Anonymous memory is reworked on a lazy foundation: sys_mmap only records the region, while page tables, zeroing and host-memory commit are deferred to first touch. Two optimization layers sit on top.
mmap: the common shape (addr == 0, MAP_PRIVATE|MAP_ANONYMOUS, RW) is served entirely at EL1 with no vCPU exit and no lock. The shim stays stateless: it bump-allocates from a host-carved per-vCPU VA arena and reports each grant through a per-vCPU SPSC ring that the host drains whenever it takes mmap_lock, so no cross-vCPU synchronization exists. Arenas recycle freed address space through the gap allocator, refill automatically when a request does not fit (the new arena sized to cover the triggering request, with an ARENA_MAX guard so oversized requests fall back without churning the arena), and decay back to workload scale based on served-length history.
Page fault: a per-2MiB-block dirty map lets materialization skip zeroing blocks whose slab bytes are known zero, moving the host page commit off the fault and out of mmap_lock onto the guest's own first writes. The TLBI protocol emits a single range op per materialized block instead of per-page invalidations, and a repeat fault on an already-valid block returns without re-zeroing.
Host-side access to untouched lazy memory (I/O buffers, futex words, signal frames, shmat/shmdt copies) materializes on demand with lock-order-safe pre-faulting, and PROT_NONE stays a pure faulting reservation. tests/test-mmap-lazy.c and tests/test-mmap-fastpath.c pin the contract: zero-on-reuse, huge sparse reservations, host access to untouched memory, PROT_NONE round trips, fork, racing first touch, and arena refill/revocation.
Close #165
Summary by cubic
Implements lazy anonymous mmap with an EL1 fast path for common anon RW calls and EL1 deferred munmap. Cuts mmap/munmap latency, reduces contention, and fixes mremap edge cases. Closes #165.
-
New Features
- Lazy anon memory with a per‑2 MiB dirty map: skip zeroing clean blocks, batch TLBI per block, ignore repeat faults; bootstrap marks ELF/shim/stack blocks dirty to preserve zero‑on‑reuse on MAP_FIXED.
- EL1 mmap fast path for
mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, ...)up to 32 GiB: per‑vCPU VA arenas + SPSC rings with adaptive refill and automatic fallback. - EL1 deferred
munmap: retire with broadcast TLBI and defer host cleanup; drain on VM‑exit and onmmap_lockacquire; per‑2 MiB PTE‑present index accelerates teardown and arena recycle. - Host access pre‑faulting for futex words, signal frames,
execargv/env copying, and SysV SHM;PROT_NONEremains a pure reservation. - Robust
mremap: preserve neighbor PTEs on cross‑block extend and fully roll back destination on ENOMEM; new tests cover splits, grow/move, and adjacent‑page safety. - Timing and tooling: enable CNTVCT reads on worker vCPUs; add
bench-mmapandbench-mmap-lazy, plus fast‑path/dirty‑map stats scripts and counters (incl. TLBI wire mode).
-
Migration
- EL1 mmap/munmap fast paths are on by default and fall back when shape unsupported, arenas exhausted, or rings full. Set
ELFUSE_MMAP_FASTPATH=0to disable; verbose tracing, the syscall histogram, GDB, and Rosetta force the host path. - Fork IPC protocol bumped to ELFM. Update both parent and child when mixing versions.
- EL1 mmap/munmap fast paths are on by default and fall back when shape unsupported, arenas exhausted, or rings full. Set