Optimize anonymous mmap and page-fault paths

Draft
MA
2mo ago
+7717-265

Anonymous memory is reworked on a lazy foundation: sys_mmap only records the region, while page tables, zeroing and host-memory commit are deferred to first touch. Two optimization layers sit on top.

mmap: the common shape (addr == 0, MAP_PRIVATE|MAP_ANONYMOUS, RW) is served entirely at EL1 with no vCPU exit and no lock. The shim stays stateless: it bump-allocates from a host-carved per-vCPU VA arena and reports each grant through a per-vCPU SPSC ring that the host drains whenever it takes mmap_lock, so no cross-vCPU synchronization exists. Arenas recycle freed address space through the gap allocator, refill automatically when a request does not fit (the new arena sized to cover the triggering request, with an ARENA_MAX guard so oversized requests fall back without churning the arena), and decay back to workload scale based on served-length history.

Page fault: a per-2MiB-block dirty map lets materialization skip zeroing blocks whose slab bytes are known zero, moving the host page commit off the fault and out of mmap_lock onto the guest's own first writes. The TLBI protocol emits a single range op per materialized block instead of per-page invalidations, and a repeat fault on an already-valid block returns without re-zeroing.

Host-side access to untouched lazy memory (I/O buffers, futex words, signal frames, shmat/shmdt copies) materializes on demand with lock-order-safe pre-faulting, and PROT_NONE stays a pure faulting reservation. tests/test-mmap-lazy.c and tests/test-mmap-fastpath.c pin the contract: zero-on-reuse, huge sparse reservations, host access to untouched memory, PROT_NONE round trips, fork, racing first touch, and arena refill/revocation.

Close #165


Summary by cubic

Implements lazy anonymous mmap with an EL1 fast path for common anon RW calls and EL1 deferred munmap. Cuts mmap/munmap latency, reduces contention, and fixes mremap edge cases. Closes #165.

  • New Features

    • Lazy anon memory with a per‑2 MiB dirty map: skip zeroing clean blocks, batch TLBI per block, ignore repeat faults; bootstrap marks ELF/shim/stack blocks dirty to preserve zero‑on‑reuse on MAP_FIXED.
    • EL1 mmap fast path for mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, ...) up to 32 GiB: per‑vCPU VA arenas + SPSC rings with adaptive refill and automatic fallback.
    • EL1 deferred munmap: retire with broadcast TLBI and defer host cleanup; drain on VM‑exit and on mmap_lock acquire; per‑2 MiB PTE‑present index accelerates teardown and arena recycle.
    • Host access pre‑faulting for futex words, signal frames, exec argv/env copying, and SysV SHM; PROT_NONE remains a pure reservation.
    • Robust mremap: preserve neighbor PTEs on cross‑block extend and fully roll back destination on ENOMEM; new tests cover splits, grow/move, and adjacent‑page safety.
    • Timing and tooling: enable CNTVCT reads on worker vCPUs; add bench-mmap and bench-mmap-lazy, plus fast‑path/dirty‑map stats scripts and counters (incl. TLBI wire mode).
  • Migration

    • EL1 mmap/munmap fast paths are on by default and fall back when shape unsupported, arenas exhausted, or rings full. Set ELFUSE_MMAP_FASTPATH=0 to disable; verbose tracing, the syscall histogram, GDB, and Rosetta force the host path.
    • Fork IPC protocol bumped to ELFM. Update both parent and child when mixing versions.