Keep guest rlimits out of the host process

Closed
XA
9d ago
+672-245

Intent: a guest setrlimit or prlimit64 changes a guest table and never reaches host setrlimit. This PR stores the limits; it does not enforce them.

Failure fixed: on main, every limit but NOFILE goes to host setrlimit, so the limit lands on elfuse itself:

  • A guest that sets RLIMIT_FSIZE to 4 KiB and writes 8 KiB kills elfuse with SIGXFSZ (exit 153), even when the guest ignores SIGXFSZ. On Linux the write returns 4096 and the next write fails with EFBIG.
  • A guest that sets RLIMIT_CPU to 1 s and spins stalls the vCPU until the watchdog reports "vCPU execution timed out after 10s", and elfuse then hangs. On Linux the guest gets SIGXCPU after 1 s.
  • setrlimit(RLIMIT_AS, 64 MiB) and setrlimit(RLIMIT_DATA, 1 MiB) fail with EINVAL from the host. Linux accepts both.
  • RLIMIT_RSS overwrites RLIMIT_AS, RLIMIT_LOCKS through RLIMIT_RTTIME are EINVAL, and only NOFILE crosses fork.

So the enforcement main has for FSIZE and CPU is already wrong: it acts on the host process. This PR removes it and does not yet put the Linux enforcement in its place.

This PR: all sixteen limits live in a guest table, seeded from host getrlimit at boot, carried in the fork IPC header and kept across execve. /proc/self/limits prints the table. NOFILE stays enforced as before. prlimit64 reaches only the caller's own process (pid 0, its pid, or one of its thread ids). Any other pid is ESRCH. For a pid that doesn't exist, that matches Linux. For a live fork-family process, Linux would act on that process; that is follow-up 4. On main, the pid is ignored, so prlimit64(child, ...) changes the caller's own limits.

Follow-ups, each enforcing from this table:

  1. RLIMIT_FSIZE on write, truncate, fallocate, sendfile, copy_file_range and splice, with SIGXFSZ to the thread that wrote.
  2. RLIMIT_CPU from sampled process CPU time, with SIGXCPU.
  3. RLIMIT_AS and RLIMIT_DATA on brk, mmap, mremap and mprotect.
  4. prlimit64 on another fork-family pid, with the kernel's credential check.
  5. RLIMIT_STACK, by growing the stack on demand, which also sizes what execve accepts as arguments.

Depends on: none.

Review focus:

  • The fork IPC header now carries the table in place of the two NOFILE fields, so the protocol magic moves from ELFQ to ELFR (src/runtime/fork-state.h).
  • sys_rlimit_cur reads a slot with no lock; writers hold rlimit_lock (src/syscall/sys.c).
  • rlimit_exchange follows do_prlimit: raising a hard limit takes guest euid 0, and the NOFILE hard limit cannot rise above the fd table size.
  • The host stack hard limit at macOS's MAXSSIZ cap seeds the guest hard limit as unlimited, since that cap bounds host threads, not guest memory.
  • ELFUSE_HOST_RLIMIT_CHECK is a test hook: with it set, a process exits 86 when a host limit moved during the run. test-rlimit-unchanged-host runs a guest under it.

Verified on b9c5e67, rebased onto b5e3fe8: make check-format and make check pass; make test-matrix-elfuse-aarch64 gives 300 passed, 0 failed, 10 skipped; make test-matrix-qemu-aarch64 gives 278 passed, 0 failed, 32 skipped.

Summary by cubic

Keeps guest setrlimit/prlimit64 calls out of the host process: limits now live in a guest table seeded from host getrlimit at boot, carried across fork and execve, and printed by /proc/self/limits from one snapshot. The old path forwarded every limit but NOFILE to host setrlimit, which let a guest FSIZE limit kill elfuse with SIGXFSZ, a CPU limit stall the vCPU, and returned EINVAL for AS/DATA; RSS also overwrote AS, which macOS aliases. All sixteen limits are stored, but only NOFILE is enforced; FSIZE, CPU, and AS/DATA enforcement is follow-up work.

Behavior changes

  • Fork IPC carries the whole table in place of the two NOFILE fields, moving the protocol magic to ELFR.
  • prlimit64 reaches only the caller's process (pid 0, own pid, or own tid); any other pid gets ESRCH, where main silently applied the call to the caller.
  • Raising a hard limit requires guest euid 0, and macOS's MAXSSIZ stack cap seeds the guest hard stack limit as unlimited.

Testing

  • ELFUSE_HOST_RLIMIT_CHECK makes elfuse record its host limits at start and exit 86 if any moved; the new test-rlimit-unchanged-host runs test-proc-limits under it.
  • test-proc-limits grows to cover all-limit round trips, fork/execve carry-over, and the RSS/AS split.