Synthesize /dev/bus/usb and the USB sysfs tree

Merged
JO
2mo ago
+5413-36

[!NOTE]
Second piece of the general USB translation layer discussed in #319, sitting on top of the netlink piece merged as #325. With #325 alone a libusb guest reaches libusb_init = 0 but get_device_list still returns 0; this piece is what puts devices in that list. It is read-only enumeration, so it stands alone and does not depend on the pieces that follow.

Summary

libusb and nusb do not discover devices through a syscall. They read Linux sysfs directly: /sys/bus/usb/devices/<bus>-<port> for the attribute files, and /dev/bus/usb/BBB/DDD for the raw descriptor blob. A guest running on elfuse has neither tree, so enumeration came back empty no matter how far init got.

This change synthesizes both trees from an IOKit registry walk. No device is opened to build them: registry properties supply the 18-byte device descriptor, and GetConfigurationDescriptorPtr supplies the raw configuration descriptors without an open. An unmodified guest binary now enumerates the attached hardware.

What

New src/runtime/usb-sysfs.c materializes both trees as scratch directories of real host files, following the ensure_syscpu_dir pattern already in procemu.c:

/sys/bus/usb/devices/<bus>-<ports>          device attribute dirs
/sys/bus/usb/devices/<bus>-<ports>:<c>.<i>  interface attribute dirs
/dev/bus/usb/BBB/DDD                        char 189:(bus-1)*128+(dev-1)

Design decisions that are worth a reviewer's attention:

Bus numbering. busnum is the top byte of the IOKit locationID plus one. macOS controller indices start at 0 while Linux busnums start at 1, so a literal locationID >> 24 would produce a bus 0 that no Linux tool expects. The port path below it comes from the remaining locationID nibbles, six at most, which is the same <bus>-<port>.<port> spelling linux_usbfs.c parses.

devnum. Taken from the USB Address property. Devices without that property get a stable per-bus fallback, numbered 1..n in locationID order and skipping numbers already taken, so a device never silently collides with another and the numbering does not shuffle between reads within a run.

Root hubs are absent, on purpose. macOS publishes no root hub as a USB device, verified with ioreg: there is no IOUSBDevice entry to name. So the tree carries no usbN entry and no parent link from a device up to its bus. Enumeration is unaffected, because libusb derives topology from the <bus>-<port> names alone and nusb drops root hubs outright. The nports == 0 branch in the code is not a filter for root hubs, it is the guard for an entry with no port path to name it by, and the comment says so. The one visible consequence is lsusb -t, covered below.

One descriptor blob, two readers. The descriptors sysfs attribute and a read() of the matching /dev/bus/usb node are served from the same buffer, so the two views cannot drift. usb_sysfs_descriptors_dup exports it for the usbdevfs fd that comes next. The test asserts the two reads are byte-identical rather than merely both parseable.

One view of the active configuration. bNumInterfaces, bmAttributes, bMaxPower, configuration, and the emitted :c.i interface directories all read the current configuration through a single active_config() helper. An earlier revision read bNumInterfaces from the first configuration in the blob while the interface directories came from the active one, which disagreed on any device whose active configuration is not the first. That class of bug is now structurally impossible, and the test pins bNumInterfaces against the actual directory count.

All four attributes exist on every device, whatever the blob held. Linux keeps configuration, bNumInterfaces, bmAttributes and bMaxPower in dev_attr_grp (sysfs.c:782-786), an attribute group with no .is_visible hook (sysfs.c:815-817), so every USB device directory carries all four files. When udev->actconfig is NULL, usb_actconfig_show returns the 0 its lock handed it (sysfs.c:29-43) -- a zero-length read, not an error. Emitting an attribute only when its value was known would turn that into ENOENT, and the two are different answers to a reader probing for an optional attribute: absent means unsupported, empty means supported with nothing to say. libudev makes the distinction observable -- udev_device_get_sysattr_value() returns a pointer to "" for an empty attribute and NULL for an absent one. So all four now render through one pure leaf, usb_desc_actconfig_attrs, and the caller writes all four unconditionally; presence is structural rather than repeated per line. (Contrast manufacturer/product/serial, which are gated by dev_string_attrs_are_visible and which this tree correctly omits when absent.)

SYSFS_MAGIC. statfs and fstatfs report 0x62656572 for /sys paths and for descriptors fds. libudev checks this before it will trust the tree, so without it the whole tree reads as untrustworthy to a udev-backed consumer.

Path handling. . and .. inside a /sys path are resolved rather than refused, so /sys/bus/usb/devices/../devices resolves the way it does on Linux instead of reporting EACCES; a suffix that climbs above /sys is handed back to the normal path flow. The lexical fold decides only whether a name is this layer's at all. The name that is actually served is resolved component by component, so a .. behind a symlink pops the target's parent the way the kernel does: <dev>/subsystem/.. is /sys/bus, which makes <dev>/subsystem/../usb/devices resolve and <dev>/subsystem/../idVendor ENOENT -- the reverse of what folding the name first produces. st_ino is derived from the resolved spelling, so two ways of naming one directory agree on their identity and an (st_dev, st_ino) same-file test tells the truth. A node addressed as a directory, a bare trailing slash included, reports ENOTDIR once the node exists and ENOENT when it does not. Symlinks inside the tree are followed the way Linux follows them, so open() of a subsystem link reaches the target directory and libudev's and pyserial's USB-backed checks resolve. Every link in the tree is written by this code with a fixed target and the tree is read-only to the guest, so following is safe; containment is proved rather than assumed, in that the resolved path is canonicalized and tested against the canonical scratch root and anything landing outside reports ENOENT instead of being opened. open, stat, lstat and readlink all resolve through that one helper, so the entry points cannot disagree about what the tree holds -- readlink opens nothing, but a target string is still the guest learning about a host object it was never handed. That property did not hold on the first revision and the review caught it: O_NOFOLLOW split the final component off, contained the prefix, then reattached the leaf, so a .. leaf reattached to the canonical root named the host directory the tree sits in. open("<dev>/subsystem/../../..", O_NOFOLLOW) returned a descriptor on the host temp parent while stat, lstat and readlink of the same name refused it and logged the refusal. A . or .. leaf is no longer held back, since neither can be a symlink and O_NOFOLLOW has nothing to protect there, and containment is now factored into one usb_sys_contained() applied last on every path, including after the reattach. A probe that plants a file in the backing directory and asks 16 names through open, O_NOFOLLOW open, stat, lstat, readlink and openat reports 4 escapes against the first revision and 0 now, with all entry points agreeing on every name. O_NOFOLLOW on a symlink reports ELOOP before the access mode is consulted, which is the order Linux uses (open(subsystem, O_WRONLY|O_NOFOLLOW) is ELOOP, plain O_WRONLY is EISDIR, measured on 6.19), and O_PATH | O_NOFOLLOW yields an fd on the link itself.

Descriptor identity, and the errnos around it. A descriptor on a followed subsystem link names the directory it resolves to, not the spelling the guest opened, so openat(fd, "..") pops the target's parent. That is what Linux does (open("/sys/class/net/lo/subsystem", O_PATH) names /sys/class/net and .. off it names /sys/class, measured on 6.19) and the stamp now comes from F_GETPATH on the descriptor rather than from the request, which needs no special case because F_GETPATH already reports the target for a followed open and the link for an O_SYMLINK one. O_DIRECTORY|O_CREAT is EINVAL, decided while the open flags are built and therefore ahead of path resolution and of the dirfd check, with O_PATH excluded because the kernel masks the creation bit off there and O_PATH|O_DIRECTORY|O_CREAT opens the directory. Every sysfs attribute is poll-capable, not just the CPU subtree, because kernfs_fop_poll is installed by sysfs_file_operations on all of them, so epoll_ctl(ADD) on <dev>/uevent answers 0 rather than EPERM; udev-style readers arm uevent before they read it. statfs decides which filesystem answers from what the name resolves to, so /sys/../sys and a relative sys from a cwd of / both report SYSFS_MAGIC. st_rdev is composed in uint32_t before it becomes a dev_t: dev_t is a signed 32-bit int on macOS and shifting major 189 left by 24 lands in the sign bit, which UBSAN reports as undefined. The ENOENT rebase fallback that lets chase() step from a host directory into a synthetic subtree now requires the descriptor to be inside the sysroot, so a sysroot symlink pointing out of the tree cannot hand a host /dev dirfd to the /dev/bus/usb intercept.

Interface numbers are a whole byte. bInterfaceNumber is a __u8, so the selection is sized to all 256 values rather than 32; the earlier bound silently dropped interfaces numbered 32 and above and left bNumInterfaces disagreeing with the directories on disk. The walk itself is bounded by cfg_len, so the table size was never what kept it in bounds.

Read-only for now. O_RDONLY on a node serves the descriptor blob; O_RDWR and O_WRONLY report EACCES with a one-line warning. Write access to a directory is refused the way open(2) refuses it, with EISDIR rather than a readable descriptor or an EACCES: /dev/bus/usb, the per-bus directories and /sys/bus/usb/devices all answer alike, while a create under the read-only /sys tree still answers EACCES. The directory opener is shared with /proc, which had the same gap, so that spelling is corrected too. The typed usbdevfs fd and its ioctls are the next piece; this one deliberately carries no transfer path.

Layout deviations that are deliberate, and their cost. The /sys/bus/usb/devices entries are real directories rather than symlinks into /sys/devices/..., so realpath() of an entry canonicalizes to itself. libusb opens attributes relative to the entry and nusb canonicalizes the entry path, so both tolerate it. Two smaller ones are accepted rather than hidden, and both are recorded in docs/internals.md:

  • the /dev/bus/usb device nodes are placeholder files, so getdents64 reports them with d_type DT_REG where Linux reports DT_CHR. stat() is correct -- the intercept fills S_IFCHR with major 189 and the right minor, which is what libusb, nusb and lsusb consult -- so only a scanner that filters on d_type alone and never calls stat would skip them. No such consumer is known, and creating real character devices needs privileges elfuse does not ask for.
  • statfs on /dev/bus agrees with fstatfs but reports the scratch filesystem's magic instead of DEVTMPFS_MAGIC.
  • configuration is always empty, and this is the one deviation that is one-directional rather than cosmetic. The attribute holds the string named by the active configuration's iConfiguration, and a string descriptor is fetched over an ep0 control transfer on an open device -- which this piece deliberately does not do. IOKit offers no way around it, and that was checked against the live registry rather than assumed: an IOUSBHostDevice entry publishes iManufacturer, iProduct and iSerialNumber together with the three strings the family caches for them (USB Vendor Name, USB Product Name, kUSBSerialNumberString), but no iConfiguration index and no configuration string. An empty file is nevertheless the honest answer rather than a missing one, because it is the answer Linux gives from the same position: configuration_show emits nothing whenever actconfig->string is NULL (sysfs.c:83-84), and usb_cache_string is documented to return NULL "if the index is 0 or the string could not be read" (message.c:1074-1075). An empty configuration on Linux therefore means "no cached string", which is strictly weaker than "iConfiguration is 0" -- a device that NAKs its own string descriptor reads empty on real sysfs too, and every device here takes that second route. The residual gap: a configuration whose iConfiguration is non-zero and whose string is readable shows that string on Linux and shows nothing here, until an ep0 path exists to fetch it.

Nothing in scope reads either, and I would rather name them than have them found. A third one is gone rather than documented: stat() of a subsystem link now reports the directory it resolves to and lstat() reports the link, because the stat intercept takes the follow flag its callers already had.

Snapshot. The tree is built once per run, so a replug is not visible within a run until the uevent layer can invalidate it. Documented, not papered over.

No device ceiling. The device table grows on demand rather than stopping at a fixed 64. usbfs itself caps only at 127 devices per bus, and a chain of hubs can put more than any fixed guess on one machine; a truncated model would lose devices from both the /dev/bus/usb and the /sys view at once. Only a failed allocation stops the walk, and it says so.

O_PATH dirfds. An O_PATH descriptor on a non-directory answers ENOTDIR for a relative name resolved against it, as openat(2) does. The empty path is excluded from that rule, so fstat() of such a descriptor still reports the object it names.

Link and plumbing. The Makefile adds -framework IOKit -framework CoreFoundation and the new source. path.c gates claim /sys (minus the existing syscpu stub) and /dev/bus for the open and stat intercepts; openat/fstatat/readlinkat on synthetic paths resolve through path_rebase_hostdirfd and are re-offered to the intercepts, which is what keeps systemd chase()'s per-component walk on the synthetic tree. usb_lock is documented as a leaf.

Tests. New tests/test-usb-sysfs.c with its own make check lane, in two halves. The first needs no device at all (SYSFS_MAGIC from statfs and fstatfs alike, the read-only open contract, the .. fold, the ENOENT/EINVAL/ENOTDIR answers), so a machine with an empty bus still covers the path layer. The second walks whatever is attached and asserts the identities that need a device. It prints the device count, so a run that covered only the first half says so instead of reading as full cover. Writing it found two bugs before review did: the bare trailing slash returning a descriptors fd, and the unfolded st_ino. The descriptor blob is device-supplied, so the walks over it live in a pure leaf unit, src/runtime/usb-desc.c, with tests/test-usb-desc-host.c covering the malformed cases natively on the host: a bLength past the end of the buffer, a zero bLength, a truncated trailing descriptor, a wTotalLength longer than the blob, and a wTotalLength shorter than the configuration it names -- which without a header check lands the cursor on an interface descriptor and returns it as the active configuration, since an interface's byte 5 sits where bConfigurationValue does. The iterator compares in the remaining-bytes domain, so a bad length stops the walk in bounds instead of stepping the cursor off the end.

The same unit covers the active-configuration attribute set, which is where the interesting branches are hardware-inaccessible: no active configuration at all; a cfg_len below the header size; iConfiguration 0 with and without a string offered (one offered at index 0 is dropped, because usb_cache_string short-circuits before any transfer); iConfiguration non-zero with the string unreadable (NULL and "" alike); iConfiguration non-zero with the string readable, which is the branch no attached device can reach; a string longer than the kernel's MAX_USB_STRING_SIZE, which is truncated in its own tail and never in the trailing newline sysfs_emit always writes; and a two-configuration blob where selecting the wrong one shows the wrong string. That leaves only the emit-to-file step hardware-conditional, and test-usb-sysfs.c covers that against whatever is attached.

Three of the device-half assertions were rewritten because they could not fail. bmAttributes, bMaxPower and configuration are now checked against the device's own descriptors blob -- strtol's endptr against cfg[7], the power string against cfg[8] scaled by the speed's unit, and configuration empty whenever iConfiguration is 0 -- through a second, deliberately independent walk of the blob inside the test, so the test cannot pass by agreeing with the code under test. A mutant that emits the wrong bmAttributes byte, swaps the power unit and writes a non-empty configuration passes the old assertions 56/0 and fails the new ones in 5 places. A containment assertion was added for the same reason, and the review found that it too could not fail: the probe path dropped the tmp component the .. chain re-entered, so it never named the plant and passed on ENOENT while the resolver was leaking. It now asserts the chain with nothing appended, pins each depth by st_dev/st_ino against /sys/bus and /sys rather than by errno (refusing every depth would satisfy "not an escape" while breaking every relative walk through the link), takes the plant appearing in the opened directory's getdents listing as the oracle, and reads the plant back by its own name first so the refusals cannot both be reporting an absent file. Against the first revision it reports 100 passed, 33 failed, naming the escaped path and the planted file. The probe name comes from mkstemp, because the guest pid is always 1 and a pid suffix separates no lanes; eight concurrent lanes now pass 8/8 where the fixed name gave 3/8. The blob reader grows its buffer instead of capping: a fixed cap answered "the file was this long" and "the read stopped here" with one number, and the two-view compare read it as a length, so two blobs differing past the cap compared equal.

Evidence

Clean rebuild at this tip, ESP32-S3 (303a:1001) attached behind a Fresco Logic hub.

lsusb-lite, a static guest binary that walks sysfs exactly the way linux_usbfs.c does and then byte-compares the sysfs blob against the /dev/bus/usb node:

STEP statfs_/sys = 0 (0 ok) f_type=0x62656572 SYSFS_MAGIC
STEP opendir_/sys/bus/usb/devices = ok (0 ok)
DEVICE 2-1
  realpath = /sys/bus/usb/devices/2-1
  busnum = 2
  devnum = 1
  idVendor = 303a
  idProduct = 1001
  bcdDevice = 0101
  version =  2.00
  bDeviceClass = ef
  bDeviceSubClass = 02
  bDeviceProtocol = 01
  speed = 12
  bConfigurationValue = 1
  manufacturer = Espressif
  product = USB JTAG/serial debug unit
  serial = 34:85:18:42:6C:98
  descriptors = 116 bytes
  IFACE 2-1:1.1: bInterfaceNumber=01 bInterfaceClass=0a bInterfaceSubClass=02 bInterfaceProtocol=00
  IFACE 2-1:1.0: bInterfaceNumber=00 bInterfaceClass=02 bInterfaceSubClass=02 bInterfaceProtocol=00
  IFACE 2-1:1.2: bInterfaceNumber=02 bInterfaceClass=ff bInterfaceSubClass=ff bInterfaceProtocol=01
  node /dev/bus/usb/002/001 = 116 bytes -> MATCH
  PARSE OK
RESULT devices_ok=2 sysfs_ok=1

The hub enumerates too (1-1, Fresco Logic 1d5c:5801, MATCH), trimmed here for length.

The five attributes lsusb -t warns about, plus bMaxPower, read back on both devices:

== /sys/bus/usb/devices/1-1 ==        == /sys/bus/usb/devices/2-1 ==
  bmAttributes   = [e0]                 bmAttributes   = [c0]
  bMaxPower      = [0mA]                bMaxPower      = [500mA]
  configuration  = []                   configuration  = []
  maxchild       = [0]                  maxchild       = [0]
  rx_lanes       = [1]                  rx_lanes       = [1]
  tx_lanes       = [1]                  tx_lanes       = [1]

Those are not free-standing values; they are the configuration descriptor decoded. Bytes 18..26 of 2-1/descriptors are the configuration descriptor itself:

09 02 62 00 03 01 00 c0 fa
                  ^^ ^^ ^^
                  |  |  bMaxPower 0xfa = 250 units = 500mA
                  |  bmAttributes 0xc0 = bus-reserved | self-powered
                  iConfiguration 0, so no string is named and the
                  configuration file is empty -- as it is on Linux

bNumInterfaces is the 03 in that same descriptor, and three :1.N directories exist. configuration being empty is the Linux behavior for a configuration whose string was not cached, not a stub -- see the deviation note above for why the file is present and empty rather than omitted. maxchild is 0 for both, including the hub, because the hub class descriptor needs an open device; the code says so at the write site.

The one-blob invariant, and the node's identity:

$ cmp /sys/bus/usb/devices/2-1/descriptors /dev/bus/usb/002/001 && echo IDENTICAL
IDENTICAL
$ cat /sys/bus/usb/devices/2-1/dev
189:128

189:128 is (bus-1)*128 + (dev-1) for bus 2 device 1, and uevent agrees (MAJOR=189 MINOR=128 DEVNAME=bus/usb/002/001 PRODUCT=303a/1001/101 TYPE=239/2/1).

Test lane and full suite on a clean rebuild:

test-usb-sysfs: 138 passed, 0 failed - PASS
  devices examined: 2
test-usb-desc-host: all tests passed
test-sysroot-symlink-escape: 9 passed, 0 failed - PASS
make check BAREMETAL_CROSS=aarch64-elf- exit=0
  All 89 tests passed
  Results: 7 passed, 0 failed, 0 skipped (of 7)
  Results: 81 passed, 0 failed, 3 skipped (of 84)
  Results: 27 passed, 0 failed, 0 skipped (of 27)
  Results: 4 passed, 0 failed, 0 skipped (of 4)
make check-ubsan exit=0   0 runtime errors, test-usb-sysfs 138/0
make check-asan  exit=0   0 sanitizer reports, test-usb-sysfs 138/0

test-usb-sysfs moved onto the sanitizer lane through CHECK_SHARED_LANES rather than tests/manifest.txt: .ci/check-matrix-lists.sh rejects a manifest binary the matrix never runs, confirmed by adding it and watching the gate fail. It belongs there because this layer composes a dev_t by shifting a major number, which is exactly the arithmetic UBSAN is there to adjudicate, and nothing else on the lane builds the tree.

Run against the previous revision of this branch, the same test reports:

test-usb-sysfs: 100 passed, 33 failed - FAIL
escaped: /sys/bus/usb/devices/2-1/subsystem/../../.. lists the planted elfuse-usb-escape-probe-shftDx

(The three skips are environment-driven and identical on main: two musl fixtures absent, one busybox applet not in this build.)

On the self-hosted runner, which has four devices where this board has two, and at least one whose active configuration carries a non-zero iConfiguration:

  configuration is empty when iConfiguration is 0            OK   (x4)
  the four active-config attributes are all present          OK   (x4)
  an absent device attribute is ENOENT, not EACCES           OK   (x4)
  devices examined: 4
test-usb-sysfs: 126 passed, 0 failed - PASS

What does not work yet, stated plainly. Real Debian lsusb (usbutils 018 over libusb-1.0.28, the udev-backed build) still does not reach this tree:

$ elfuse --sysroot ./sysroot-libusb /usr/bin/lsusb
WARN unimplemented syscall 264 (...)
unable to initialize libusb: -99

Syscall 264 is name_to_handle_at, which libudev uses to decide whether /sys is a genuine sysfs mount. That is a separate change and not in this PR, so the evidence above comes from a static guest binary that walks the same sysfs paths libusb walks, not from lsusb itself. Transfers are also not here: this piece is enumeration only, and reading or writing an endpoint needs the usbdevfs fd that follows.

Known boundary: lsusb -t

lsusb -t lists devices without the bus rows above them. The README feature line now says so, and no longer claims the tree is enough for lsusb at all: a udev-backed lsusb needs name_to_handle_at as well, which is not implemented yet, so the bullet promises libusb-style enumeration and names both gaps. lsusb -t no longer prints the per-device warnings either: the five missing-attribute warnings per device are gone, and it exits 0 silently. That is the improvement this PR makes to tree mode, and it is the whole of it.

The empty tree itself is structural and recorded in #319 (comment). Tree mode does not enumerate through libusb; it walks /sys/bus/usb/devices itself and roots the tree at the usbN root-hub entries, which macOS gives us nothing to build from. Making the tree render is a coherent chunk of work rather than a missing attribute: usbN entries occupy Device 001 on each bus on Linux, so real devices would renumber to 002 and up, moving their /dev/bus/usb paths, and the per-interface driver symlinks would then have to agree with what GETDRIVER reports. No consumer in scope reads any of it, so it is recorded as a known boundary rather than planned work. If a real workflow needs tree mode, it can become its own issue then.