Keep rlimits in guest state and enforce them

XA
2h ago
+4217-508

A guest setrlimit reached host setrlimit for every resource but NOFILE, so a guest limit constrained elfuse itself and nothing was enforced the way Linux enforces it: a finite RLIMIT_CPU stopped the vCPU from making progress until the watchdog fired, RSS and AS shared one macOS slot, and only NOFILE crossed fork. This series keeps all sixteen Linux resources in a guest table with no host call behind it, carries the table through fork and execve, and makes prlimit64 reach another fork-family pid with the kernel's credential check.

The limits are then enforced where Linux enforces them: FSIZE on write, truncate, fallocate, sendfile, copy_file_range and splice, with SIGXFSZ sent to the thread that wrote; CPU from sampled process CPU time; AS and DATA on brk, mmap, mremap and mprotect; STACK by growing the stack on demand, which also sizes what execve accepts as arguments. /proc/self/limits reads the same table.

Each behavior was measured on the reference kernel first (Linux 6.18 in the qemu lane), and the tests assert what it returned, so the same test binaries pass in both lanes.

What a reviewer may want to know before the diff:

  • The fork IPC header carries the table, so the protocol magic moves from ELFQ to ELFR.
  • Each process publishes its table as elfuse-rlim-<hostpid> in the per-user temp directory, which is how prlimit64 reaches another process.
  • The stack is no longer an 8 MiB region mapped whole. The [stack-guard] record is gone and the window between the break and the stack is unmapped at execve, so tests/test-brk-stack.c changes with it.
  • ELFUSE_HOST_RLIMIT_CHECK is a test hook in the binary: with it set, a process exits 86 when a host limit has moved. test-rlimit-unchanged-host runs a guest under it.
  • Signal delivery, rt_sigaction and rt_sigprocmask copy to and from the guest with sig_lock released, in the order Linux uses, because such a copy can grow the stack and so wait for mmap_lock. A bad old-value pointer is then EFAULT with the new action or mask already in place, as in Linux.
  • RLIMIT_AS, RLIMIT_DATA and VmSize leave out [shim] and [shim-data], which a Linux process does not have. ELF segments are tracked at page granularity, as elf_map maps them, and the vDSO page is listed as [vdso], so VmSize equals what /proc/self/maps lists for the guest.

Left out: MEMLOCK, NPROC, CORE and the six resources macOS has no counterpart for (LOCKS through RTTIME) are stored and reported but not enforced. The stack cannot grow past the heap below it, about 110 MiB in the default layout, where Linux has no such bound.

Verified on c7e4572, rebased onto 8612033: make check-format, make check, make check-locks and make verify pass; make test-matrix-elfuse-aarch64 gives 302 passed, 0 failed, 10 skipped; make test-matrix-qemu-aarch64 gives 278 passed, 1 failed, 33 skipped, the failure a test-simd-clone timeout on the reference kernel that passed on three reruns.

Summary by cubic

Keeps all sixteen Linux resource limits as guest state in a single table, so a guest setrlimit no longer constrains the host process: previously every limit but NOFILE hit host setrlimit, and LOCKS through RTTIME returned EINVAL where Linux accepts all sixteen resources. The table is carried through fork and execve, prlimit64 reaches another fork-family pid with the kernel's credential check and EPERM on a mismatch, and /proc/self/limits reads the same table. A host at macOS's MAXSSIZ cap seeds the guest hard stack limit as unlimited, since the cap bounds host threads, not guest memory.

Enforcement

  • FSIZE at write, truncate, fallocate, sendfile, copy_file_range and splice, with SIGXFSZ to the thread that wrote.
  • CPU from sampled process CPU time; AS and DATA on brk, mmap, mremap and mprotect, held to DATA with a soft limit of 0 bounding data by the hard limit the way Linux does.
  • STACK grows on demand, which also sizes what execve accepts as arguments.
  • Each behavior was measured on the reference kernel (Linux 6.18 in the qemu lane) and the tests assert what it returned, so the same binaries pass in both lanes.
  • MEMLOCK, NPROC, CORE, and the six resources macOS has no counterpart for (LOCKS through RTTIME) are stored and reported but not enforced. The stack cannot grow past the heap below it, about 110 MiB in the default layout, where Linux has no such bound.

Note for reviewers

  • Fork IPC protocol magic moves from ELFQ to ELFR; the header now carries the rlimit table.
  • The stack is no longer an 8 MiB region mapped whole: the [stack-guard] record is gone and the break-to-stack gap is unmapped at execve.
  • A host read or write under the stack takes mmap_lock to grow it; callers holding a lower-ranked lock bracket the access with mmap_lock_nowait_begin/end, and make check-locks asserts none is held where a thread waits on mmap_lock.
  • A read-only ftruncate is refused before the FSIZE check; truncate reads permission and size through one descriptor, closing a rename race, and falls back to the stat size when no descriptor can be opened.
  • RLIMIT_AS and DATA refuse a request when the region table is stale; a failed stack growth unmaps the block it mapped and refuses pages a lost mapping still holds, and a fixed mapping above guest_size checks its descriptor before the limits.
  • Execve argument sizing counts the resolved path and drops the shebang argv[0]; the stack-limit test execs itself with no environment.
  • prlimit64 with a thread id of another process is ESRCH, where Linux reaches the process, and a remote table is refused unless the publisher's pid and start time match so a reused pid cannot read a stale one.
  • The DC ZVA emulation now retraps an EL1 abort taken by one of its stores, so the guest takes the fault it would have taken at EL0 instead of the VM halting.
  • ELF segments are tracked from the page holding each p_vaddr and the vDSO page is listed as [vdso], so /proc/self/maps prints page-aligned offsets and a VmSize that equals the guest mappings.
  • ELFUSE_HOST_RLIMIT_CHECK is a test hook that makes a process exit 86 when a host limit has moved, proving guest limits never touch the host.