Redesign sidecar using byte-exact guest filenames without a sidecar index

Merged
HE
3mo ago
+12539-2742

The design is being refined as the testing and simplification of the code are carried out continuously

A Linux guest names files by exact bytes. The default APFS volume matches names
case- and normalization-blind, so Foo and foo cannot coexist and a wrong-case
lookup succeeds where Linux owes ENOENT.

The previous answer, src/syscall/sidecar.c (2217 lines), stored such a name
under an opaque random token (.ef_ plus 64 bits from arc4random, retried on
EEXIST) and recorded what the token meant in a per-directory
.elfuse_case_index file. The mapping lived outside the name, and everything
below follows from that one decision.

A name was two objects, so no mutation was atomic

Creating, removing, or renaming a name had to touch the token and the index
file, which no host syscall can do together. renameat therefore ran a
hand-rolled two-phase commit: clone both directory indices as rollback
snapshots, write the index or indices to disk, call renameat, and on failure
write the saved snapshots back. The comment conceded the gap: "A failed
rollback write is the best-effort case." A crash between the index write and
the host rename left a mapping pointing at a token that had moved or did not
exist, and nothing repaired it on the next mount. Cross-directory rename took
two index locks in a fixed order, with a swapped flag deciding which went
first, so the ordinary rename path carried deadlock-avoidance logic.

It forced serialization that the filesystem does not need

Two lock layers, because POSIX advisory locks are per-process and every vCPU
thread would share one: a process-wide pthread_mutex_t across all vCPU
threads, plus an fcntl write lock on a .elfuse_case_index.lock sentinel to
serialize against other elfuse processes on the same sysroot. Two guests
creating unrelated files in one directory contended, and the process mutex
serialized name creation across every directory at once.

The index was advisory, so anything else desynchronized it

fcntl locks bind only the processes that take them. Any writer that is not
elfuse (tar, rsync, git, Finder) could add or delete entries without
touching the index, leaving rows describing tokens that were gone and files
with no row at all. In the other direction the tokens are opaque, so a sysroot
was unreadable and unmaintainable outside elfuse, and copying one preserved
the mapping only if the index files came along intact.

Reads paid for it, and the cache was itself a correctness surface

The walker visited every parent directory of every translated path, probing
each for an index. That cost dominated startup, measured at "openat 61% of
getent's 7.5 ms warm path", so it needed a 64-slot open-addressed cache
keyed on (dev, ino, mtime, ctime), last writer wins on collision. Ordering
then became load-bearing: the publish path had to invalidate the cache
before the rename, or a concurrent walker could cache "no index here" and
miss one another process had just published.

Leftovers

Three reserved names per directory (index, .tmp, .lock) had to be filtered
out of every listing and refused as guest paths. Each mutation parsed the whole
index into a heap row array and deep-cloned it for rollback, so a create in a
large directory cost work proportional to the directory. And the sidecar
resolved paths independently of path_translate_at, so two walks disagreed
about when a guest path falls through to the host.

Approach

Decide a name's on-disk spelling from the name alone. Nothing is persisted, so
nothing can desynchronize, and no lock is needed because there is no second
object to keep consistent.

  • Codec (src/syscall/casefold.c): a name containing an uppercase ASCII
    letter, a byte above 0x7F, or an escape-shaped prefix is stored as
    .ef=<payload>. What is left literal is lowercase ASCII, a fixed point of
    every transformation the volume applies, so two literal names cannot collide
    however the folding table changes. The payload has two tiers because the
    per-name limit is 255 UTF-16 code units rather than bytes: hex to 125 bytes,
    then twelve bits per symbol from 4096 CJK ideographs at U+4E00, a block that
    neither folds nor normalizes. Guest names keep all 255 bytes Linux allows.
  • Walk (src/syscall/casefold-walk.c): resolve a path one component at a
    time, asking the volume for each name as stored (getattrlistat) and
    comparing bytes, the only way to separate "exists as spelled" from "exists
    under a spelling that folded onto it". Absence is three answers, not one:
    unclaimed falls through to the host, a slot held by a differently-spelled
    sibling is ENOENT, and a non-directory component is ENOTDIR
    (path_resolution(7)).
  • One resolver: proc_resolve_sysroot_path_flags and path_translate_at
    fold into that single walk, which also backs the openat2
    RESOLVE_NO_SYMLINKS and RESOLVE_NO_XDEV walkers. Directory entries and
    the cwd decode on the way back out.
  • Sidecar deleted: index, parsers, both locks, the snapshots, both caches,
    and its five private syscall handlers, which now get the ordinary /proc,
    FUSE, and /dev/shm handling. Colliding creates never contend, because the
    spelling is a function of the name. Two elfuse processes sharing a sysroot
    need no coordination, because no shared mutable state remains.

Prior art

The same question ("the host filesystem cannot hold this name") has three
standard answers, and this branch moves elfuse from the worst to the best of
them.

  • Ask the filesystem. Cygwin took this route: managed mounts, its
    encoding scheme, were removed in 1.7 in favor of real NTFS case sensitivity
    via the obcaseinsensitive kernel flag, with per-mount posix=[0|1].
    elfuse's equivalent is --create-sysroot, which builds a case-sensitive APFS
    sparsebundle, and Docker Desktop and Lima sidestep it entirely with a Linux
    VM over ext4. It is the cleanest answer when available, which is why the
    check suite re-runs the whole name suite on that volume as its oracle. It is
    not always available: the sparsebundle is opt-in and a user may point
    --sysroot at any directory.
  • Derive the stored name from the guest name. Where Cygwin still must
    encode, for characters Windows disallows, it transposes them into the Unicode
    private use area by adding 0xf000: a pure per-character function with no
    side table. gocryptfs, EncFS, and eCryptfs are the same family, deriving the
    on-disk name from the plaintext name and a key. This branch is that answer.
  • Keep a mapping. The sidecar was here, and so is CryFS, which stores names
    inside encrypted blocks. Even gocryptfs falls back to it, but only at the
    boundary the derivation cannot cross: when an encrypted name would exceed
    255 bytes it writes a gocryptfs.longname.<hash>.name support file. That is
    precisely what elfuse's second tier avoids. Packing twelve bits per CJK
    symbol keeps the derived property across the full 255-byte guest range, so
    there is no length at which a side file reappears.

Compatibility

A sysroot written by the old encoding must be recreated. Its .ef_<token>
entries and .elfuse_case_index files decode to nothing and surface under
their literal host names. docs/usage.md says so and gives the recipe.

Tests

Two host unit lanes (codec, resolver) link exactly the code under test.
tests/casefold-vectors.h freezes the on-disk spellings in both directions,
making a format change a deliberate migration. Guest lanes cover edge shapes
(one-character sysroot, no sysroot, chdir), i18n pairs the volume folds, name
and path length budgets, host-staged spellings, a corpus read back by guest
name, and concurrency (a fork race run ten times in check, plus a manual
soak).

Two lanes are oracles rather than expectations:

  • check-name-caseexact re-runs the name suite on a case-sensitive APFS
    sparsebundle, where the volume itself enforces what the tests assert. It is
    what surfaced the ENOTDIR and vanished-path fixes.
  • test-sysroot-path-matrix enumerates addressing mode x operation x path
    shape x name class and holds each cell to path_resolution(7): absolute,
    cwd-relative, and dirfd-relative spellings of one file must agree on result,
    errno, and object. On first run it caught two bugs nobody predicted. It runs
    on the folding tmpdir, the byte-exact volume, and the qemu reference kernel.

The matrix gains ELFUSE_SKIP as the inverse of QEMU_SKIP, with
.ci/check-matrix-lists.sh rejecting a label naming no registered test or
sitting in both lists.

Summary by cubic

Redesigns filename handling with a stateless .ef= codec and a case‑exact path resolver, replacing the sidecar so Linux byte‑exact names work on case‑folding APFS without locks or per‑dir files. All path surfaces resolve and report guest bytes (symlink targets, exec, inotify, pathname AF_UNIX, directory listings, cwd); temp roots (/tmp, /var/tmp, ~/.ccache) are consistently redirected through the sysroot for creates and lookups. .. now clamps at the guest root, and openat2(RESOLVE_NO_SYMLINKS) handles deep link‑free paths correctly.

  • Bug Fixes

    • Symlinks: follow targets in the guest namespace; absolute targets resolve against the sysroot; openat2(RESOLVE_NO_SYMLINKS) returns ELOOP when a link is present.
    • Exec: translate dirfd‑relative paths; resolve PT_INTERP through the same walk with /lib/<basename> fallback; /proc/self/exe and /proc/self/fd/N report guest bytes; shm probes honor O_NOFOLLOW.
    • AF_UNIX (pathname) sockets: bind/connect/send* resolve through the sysroot with case‑exact names; over‑long paths shorten via a private namespace symlink with correct cleanup across fork/exit; getsockname/getpeername/accept/recv* decode guest paths and fix msg name lengths.
    • Inotify: watches translate through the resolver; event names decode to guest bytes; /dev/shm leaves are watched without following.
    • Path correctness: clean ENAMETOOLONG at the host path ceiling (no truncation); trailing / keeps ENOTDIR; --sysroot / parent split fixed; directory entry decoding is scoped to sysroot‑owned directories (host fallback and no‑sysroot lists stay literal); openat2(RESOLVE_NO_XDEV) re‑checks mount class after open; .. is clamped at the guest root for absolute and dirfd‑relative paths; RESOLVE_NO_SYMLINKS no longer caps components, so deep link‑free paths resolve.
    • Cwd: getcwd and /proc/self/cwd report guest bytes so the path is chdir‑able.

  • Migration

    • Sysroots written by the old sidecar format must be recreated. Existing .ef_<token> and .elfuse_case_index files appear as literal host entries. See docs/filenames.md and docs/usage.md.