Run usbdevfs URBs asynchronously over IOKit

Merged
JO
1mo ago
+11752-585

NOTE: piece D of the USB layer described in #319, on top of #366 (piece C, the usbdevfs descriptor and its synchronous ioctls) and through it on #334 and #351. #366 answered ENOTTY for SUBMITURB / REAPURB / DISCARDURB and reported GET_CAPABILITIES as 0; this piece implements them and raises the two bits it can honour. The tty aliasing piece follows and is not here. Two commits: the engine, then the loopback fixture that tests it.

Summary

SUBMITURB, REAPURB, REAPURBNDELAY, DISCARDURB, poll/select/epoll on a usbdevfs fd, and CAP_REAP_AFTER_DISCONNECT's disconnect drain, over IOUSBDeviceInterface650 / IOUSBInterfaceInterface800 async completions delivered on one lazily-started CFRunLoop host thread -- libusb's own darwin model. GET_CAPABILITIES goes from 0 to 0x11.

The engine is verified three ways: a native host binary over the arithmetic (781 assertions), a new IOKit loopback fixture that lets the whole engine run without a board (44 assertions, plus the entire #366 fd-contract lane re-run against a device that can actually complete a transfer), and the attached ESP32-S3 for everything the fixture cannot show. All numbers below are from runs made while writing this description.

The two commits

  1. Run usbdevfs URBs asynchronously over IOKit -- the engine. SUBMITURB, REAPURB, REAPURBNDELAY, DISCARDURB, the per-endpoint queue, the disconnect drain, poll/select/epoll, and the native host binary over the arithmetic. 16 files, 2968 insertions, 182 deletions. No fixture file exists in the tree at this commit, and it builds, passes every gate, and passes make check there.
  2. Model a loopback USB device behind the IOKit seam -- the device model, the stub that stands in for it, the build variable that picks between them, the two lanes that run it, and the docs. 14 files, 2482 insertions, 34 deletions.

The boundary is the shipped binary. The model does not go into one (below), so the second commit is a translation unit no default build links plus the wiring that keeps it that way, and the engine can be read without it.

What

One transfer per endpoint in flight, and what it costs. Linux DISCARDURB cancels exactly one URB. IOKit's AbortPipe (USBDeviceAbortPipeZero for ep0) cancels every outstanding transfer on the pipe. There is no IOKit call with Linux's granularity, so the Linux semantics are kept by never giving IOKit more than one transfer per endpoint to abort: later submissions queue inside elfuse and are started from the completion callback.

The cost is real and worth stating plainly. A guest gets queue depth but no overlap: an endpoint is idle for a full host round trip between one transfer completing and the next starting, so a bulk-streaming guest that submits a ring to keep the pipe busy gets its URBs served one at a time. The ring itself is honoured -- 300 URBs queued on one fd on the board all queued (first error 0) and all 300 reaped back -- and the loopback lane asserts both halves of the price: never more than one in flight per endpoint, and the follower started only after the leader completed. Throughput on an endpoint is therefore bounded by round-trip latency, not by IOKit's pipelining. Removing the queue would restore overlap and lose DISCARDURB's one-URB granularity; that trade is the one design decision this piece is built around.

One URB on the wire is not by itself sufficient. The abort is issued with async_lock dropped, and in that window the target can complete normally and the completion callback can start the queued follower for the abort to hit instead. So the endpoint's FIFO stays shut while an abort for it runs, and DISCARDURB waits (bounded at 2s) for the record to leave the pending list -- which is also what makes an immediately following REAPURBNDELAY find the URB the way usb_kill_urb leaves it. On the board, across 63 runs, 12600 rounds of discarding one of two queued ep0 URBs produced 0 bystander ECONNRESET, and 1260 bulk-IN discards all reaped ENOENT, with the REAPURBNDELAY straight afterwards finding the URB every time.

Argument validation follows proc_do_submiturb, not do_proc_bulk. Flags mask, then USBFS_XFER_MAX, then a NULL buffer with a positive length, then the endpoint (devio.c:1631-1658). So an absent endpoint outranks an unknown transfer type, the ISO refusal and the control arm's eight-byte minimum, and a 2 GB URB on the default control pipe is EINVAL rather than an allocation. That is deliberately not the order #366 pins for the synchronous BULK ioctl, which resolves the endpoint first; the two kernel paths genuinely differ.

Resource limits are Linux's, and two of elfuse's own are gone. The 16 MB budget is the one the synchronous transfers already charge, because usbfs_memory_usage is a single kernel-wide static (devio.c:134, 143-178) and a second counter would let this path queue a budget behind whatever a BULK ioctl holds. Records are charged alongside their buffers so zero-length URBs are bounded, and there is no URB-count cap: the old per-fd 256-record backstop refused a 257th eight-byte URB, which is exactly the ring depth libusb's async API builds for bulk streaming.

The disconnect path. usbdev_remove runs destroy_all_async before it wakes the reapers. IOKit completes nothing of its own when a device terminates, so the first reap that finds the completion list empty on a disconnected fd issues the kill itself. The disconnect-watch refcon packs the slot index in a field sized from USBDEV_MAX_FDS behind a static assertion rather than a hand-written four bits: 32 slots through four bits decoded slot 16+k as slot k, which the host binary now pins -- reverting it fails 144 assertions, exactly 128 round trips plus 16 collisions.

poll/select/epoll (644 lines added to poll.c). The fd's completion pipe raises host POLLIN, remapped to guest POLLOUT|POLLWRNORM (devio.c:2830-2843); epoll registers EVFILT_READ and reports EPOLLOUT through a per-registration flag. Host-side interest is armed regardless of what the guest asked for, because a disconnect must wake even a read-only wait with the unmaskable POLLERR|POLLHUP (devio.c:2839-2842). A wake that maps to nothing guest-visible re-blocks: ppoll/pselect withdraw the woken entry's host interest (unreaped completions keep the pipe readable, so leaving it armed busy-spins at 100% CPU) and resume in bounded 200ms slices; epoll mutes the fired knote and re-enters the kevent slice loop rather than returning 0 before the timeout, which Linux ep_poll never does. read() on a disconnected fd is ENODEV (devio.c:320-322), and REAPURB returns EINTR with syscall_restart_forbid() because reap_as does not restart (devio.c:2113-2114).

The writability half of the remap reads the same derived capability the ioctl and write gates read rather than testing the access mode against O_RDONLY, matching usbdev_poll's FMODE_WRITE gate (devio.c:2837) -- access mode 3 is not O_RDONLY and carries no FMODE_WRITE either. This is offered as a consistency fix with no test, and deliberately not as a bug: no guest can observe the difference, because reaching POLLOUT needs a completion and SUBMITURB on such an fd is already EPERM, so an assertion here would pass against the literal test too.

Capability bits, and the evidence each is honoured

An earlier piece in this series was caught raising a bit it did not honour, so each bit is listed with what proves it.

  • CAP_ZERO_PACKET (0x01) -- honoured. The trailing zero-length write is emitted from the completion callback with a bounded timeout and async_lock dropped, and a failure folds into urb->status. The loopback fixture logs what crossed the seam, so the lane asserts the trailing zero-length packet reached the wire, a short OUT gets no terminating packet, and a failed terminating packet lands in urb->status -- observed on a wire, not inferred from the predicate. Wiring the predicate to false fails 2 assertions; the untimed-under-lock version reported a failed terminating packet as success.
  • CAP_REAP_AFTER_DISCONNECT (0x10) -- honoured. Loopback asserts CAP_REAP_AFTER_DISCONNECT hands back every in-flight URB, each one carries the errno usb_kill_urb leaves, and REAPURB is ENODEV once the drain is done; a reap that skips the drain fails 2. On the board, RESET returned all 3 queued URBs (rc=0, 3 of 3).
  • CAP_BULK_CONTINUATION (0x02) -- stays clear, on purpose. The flag is accepted but its error-cascade unlink is not implemented, and a guest that read the bit would rely on the cascade.
  • NO_PACKET_SIZE_LIM and BULK_SCATTER_GATHER stay clear; neither is implemented.

GET_CAPABILITIES reads 0x11 on the board in every run, and the loopback lane asserts it too.

The loopback fixture: what it is, and what it cannot show

The engine needed a device that can complete a URB, and IOKit publishes no loopback one, so ELFUSE_USB_FIXTURE=loopback is one.

The seam is exactly two IOKit COM vtables and nothing above them. Every wire call in usbdev.c goes through IOUSBDeviceInterface650 ** or IOUSBInterfaceInterface800 ** as (*h)->Method(h, ...), so src/syscall/usbdev-fixture.c hands back an object whose first member is a vtable of the same shape. Everything above stays the code that runs against a board: the URB records, the per-endpoint FIFO, usbdev_async_cb, the SHORT_NOT_OK and ZERO_PACKET predicates, the readiness and disconnect maps, REAPURB, the drain, and all 644 poll.c lines. Completions arrive from a one-shot CFRunLoopTimer on the event thread, where IODispatchCalloutFromCFMessage would have delivered them, and aborts are asynchronous the way IOKit's are.

What the fixture does is data rather than a flag: ELFUSE_USB_LOOPBACK scripts a per-endpoint sequence of outcomes, each naming the IOReturn it stands for -- short(n) kIOReturnUnderrun 0xe00002e7, stall kIOUSBPipeStalled 0xe000404f, timeout kIOUSBTransactionTimeout 0xe0004051, nodev kIOReturnNoDevice 0xe00002c0, plus ok, ok(n), err, refuse, delay(ms), never, terminate and zlpfail -- and a guest rewrites the script, reads back a log of what crossed the seam, and terminates the device through vendor control requests.

What it cannot show, stated so the coverage is not overread. Three of the engine's own answers are board-only, and breaking any of the three leaves the lane at 44 passed, 0 failed:

  1. the ZERO_PACKET write's dropped async_lock -- the fixture answers a zero-length write immediately, so nothing NAKs and nothing stalls behind the lock;
  2. its bounded timeout -- the fixture ignores both timeout arguments;
  3. the bystander window ep_aborting shuts -- the fixture retargets an aborted transfer's timer under its own lock, so it can never start a follower into an abort that is still running.

The 200-round board driver, run 63 times for 12600 rounds with 0 bystander ECONNRESET, remains the only evidence for (3). More broadly, docs/testing.md records what stays on hardware: real timing, NAKs, maxpacket segmentation, DMA alignment and throughput; that IOKit really delivers completions on the runloop, and the IODispatchCalloutFromCFMessage opacity behind the URB record's atomic owner, which a timer callout fully visible to ThreadSanitizer does not reproduce; exclusive-access arbitration and kernel-driver binding; a real SET_CONFIGURATION, altsetting renumbering and port RESET; that a device actually receives the zero-length packet; and a physical unplug mid-transfer.

None of it is in a default build. src/syscall/usbdev-fixture.c is a translation unit under src/ that exists only so a test can run, and this tree has no precedent for one. The env-var fault hooks that usb-sysfs.c, usbdev.c, fs-stat.c and fs.c carry are precedented, but each is a handful of lines inside a file that is there for production reasons; and the synthetic USB tree in usb-sysfs.c earns its place the same way -- it lets lsusb work on a machine with no devices, so a non-testing user gets something from it. A device that echoes back what was written to it does nothing for anyone outside an assertion.

So the seam has two implementations and the build picks one. src/syscall/usbdev-fixture-stub.c -- 72 lines, usbdev_fixture_loopback() answering false and the rest inert -- is in SRCS in every build; usbdev-fixture.c takes its place only when USB_LOOPBACK_FIXTURE=1 asks for it (mk/config.mk, and the SRCS block in the top-level Makefile). Listing both would be a duplicate-symbol link error, which is the property that keeps a default build from quietly acquiring the model. Not one call site in usbdev.c is conditionally compiled, so it is one program in both builds and the fixture cannot drift into code a default build never sees.

To get the fixture, ask for it:

make USB_LOOPBACK_FIXTURE=1                        # the model, in build/elfuse
ELFUSE_USB_FIXTURE=loopback build/elfuse <guest>   # and one loopback device

make check needs no flag. mk/tests.mk re-enters make with USB_LOOPBACK_FIXTURE=1 ELFUSE_BIN=build/elfuse-loopback and the two loopback lanes run that binary, so what they run is the build a reader gets from the variable and cannot drift from it, while a plain make still leaves build/elfuse without the model. Overriding ELFUSE_BIN rather than BUILD_DIR shares every object but the fixture's, so a loopback build compiles exactly one extra file, and a command-line override reaches the sub-make through MAKEFLAGS, so the sanitizer lanes get a loopback binary of their own flavor. make lint reads a new ALL_SRCS rather than SRCS, so gating the model out of the build does not gate it out of clang-tidy, and cppcheck walks the tracked sources, so it sees both either way.

What a default build carries of it is measured rather than asserted. On a clean rebuild of the tip:

build/syscall/usbdev-fixture.o        never compiled (no such file)
nm build/elfuse, fixture symbols      the 7 seam entry points, all the stub's
size -m usbdev-fixture-stub.o __text          44 bytes
size -m usbdev-fixture.o      __text       10104 bytes
defined symbols, default vs USB_LOOPBACK_FIXTURE=1     1771 vs 1811
  the 40 extra are all fx_* or fixture; none is present only in the default
the loopback binary is about 18 KB larger, the model's text being the difference

After a full make check -- which links elfuse-loopback for the lanes -- nm build/elfuse still finds zero fx_* symbols.

Off, the seam is five if (u->fake) branches, one has-device probe and one bind call, behind a mode resolved once per process and set for one modeled location only. Every other device takes the path it took before -- the existing fixture modes' descriptor blobs come out byte-identical seam-in versus seam-out. The fixture device carries no synthetic io_service_t (u->service stays IO_OBJECT_NULL and the flag says so), which keeps every IOObjectRelease and usbdev_arm_disconnect_watch's NULL-service guard correct by construction. A vtable slot usbdev.c calls that the fixture left NULL would be a null call rather than a compile error, so the list is checked once when the fixture stands up.

To show the seam changes nothing on real hardware, the four board drivers were run against the board twice: once on the engine commit, where no fixture file exists at all, and once on the tip's default build, where the seam exists and the stub answers it. The two logs are byte-identical once the wall-clock figures are normalized. Those figures jitter run to run -- the three timeouts read 5019, 5046 and 6176 ms in the first against 5082, 5033 and 6146 ms in the second -- and timing figures in this description are quoted with the run they came from and compared as jitter, never as values to reproduce.

Deviations

Eight XFAIL rows, printed by test-usbdev-ioctl with both values so they live in the lane's own output rather than only here.

XFAIL open-limit: Linux unbounded, elfuse 32 simultaneous
XFAIL reset: Linux re-enumerates the port, elfuse kills the device's URBs and
  clears claimed pipes' stalls, logging rather than reporting a clear that
  fails, and returns 0
XFAIL clear-halt-collateral: Linux warns and leaves a queued URB on the
  endpoint alone (check_reset_of_active_ep, devio.c:1379-1391), elfuse has only
  ClearPipeStallBothEnds, which aborts the pipe, so CLEAR_HALT and RESETEP make
  an in-flight URB reap -ECONNRESET
XFAIL urb-signal: Linux raises the URB's signr at completion
  (kill_pid_usb_asyncio, devio.c:654) and DISCSIGNAL's at disconnect, elfuse
  accepts both, returns 0 and delivers neither
XFAIL iso: Linux serves isochronous URBs, elfuse answers EINVAL once the
  endpoint has resolved
XFAIL discard-latency: Linux's usb_kill_urb returns with the URB already
  completed, elfuse waits for IOKit's abort callback and gives up after 2s
  rather than parking the vCPU thread
XFAIL driver-name: Linux GETDRIVER reports the driver's name (cdc_acm), elfuse
  reports the IOKit class (AppleUSBACMControl), and DISCONNECT_CLAIM's name
  filters compare against it
XFAIL short-bulk-out: Linux reports the byte count actually sent, elfuse has no
  length from WritePipeTO and reports EIO

The last three are inherited unchanged from #366. Separately, docs/usage.md now states what a stock distribution lsusb actually does here: usbutils reaches libusb through libudev, whose monitor wants SO_ATTACH_FILTER, elfuse's netlink layer answers ENOPROTOOPT, and libusb_init gives up with -99. That gap belongs to the netlink layer, not to this piece, but the worked example had to stop claiming otherwise.

Evidence

Both commits were checked out on their own, each from make clean, and each ran the whole set. The engine commit first, with no fixture anywhere in the tree:

make check BAREMETAL_CROSS=aarch64-elf-        exit 0,  All 99 tests passed
test-usbdev-urb-host                           781 passed, 0 failed
test-usbdev-ioctl        (FIXTURE=1)           103 passed, 0 failed
test-usb-sysfs / -sysroot / -overflow          138 / 13 / 5 passed, 0 failed
nm build/elfuse, fixture symbols                0
check-asan / check-ubsan / check-tsan          exit 0, 0 sanitizer reports
board: adj, full, board, conc                  4 of 4 exit 0,
                                               adj-board 7 passed, 0 failed

Then the tip, from a fully clean tree, with build/elfuse-loopback linked inside the lane that needs it:

make check BAREMETAL_CROSS=aarch64-elf-        exit 0,  All 99 tests passed
test-usbdev-urb-host                           781 passed, 0 failed
test-usbdev-urb-loopback (FIXTURE=loopback)     44 passed, 0 failed
test-usbdev-ioctl        (FIXTURE=loopback)    108 passed, 0 failed
test-usbdev-ioctl        (FIXTURE=1)           103 passed, 0 failed
test-usb-sysfs / -sysroot / -overflow          138 / 13 / 5 passed, 0 failed
check-asan / check-ubsan / check-tsan          exit 0, 0 sanitizer reports;
                                               both loopback lanes ran under
                                               each, 44/0 and 108/0 every time
make lint (clang-tidy, 63 files)               exit 0
.ci/check-cppcheck.sh (63 files)               clean

The gate set, run on each commit separately: commit-log, commentflow --check,
check-format (clang-format 22.1.8), .ci/check-cppcheck.sh, check-newline,
check-security, check-matrix-lists, check-eintr-contract, check-lock-order,
check-atomics, check-ascii, check-svc-tails, check-skill-refs,
check-proof-targets, check-syscall-coverage and make lint -- all exit 0 on
both.

The loopback run answers 108 where the FIXTURE=1 run answers 103. The five extra are exactly the in-flight transfer allowance -- with no claimable device the claim answers ENODEV before any length is looked at, so those five skip there and print why. Every assertion both runs make answers the same in both.

Negative controls, because a lane that only ever passes proves nothing:

  • Reverting each of the five things the host binary pins to the shape it had fails it: 144, 3, 3, 1 and 1 assertions (the refcon one is a compile error while the static assertion stands).
  • Of the eleven new SUBMITURB argument-order assertions in test-usbdev-ioctl, seven fail against the order the engine had first -- endpoint resolved ahead of the gate, no USBFS_XFER_MAX bound at all.
  • Ten deliberate breaks of the engine were each applied, rebuilt and run against the loopback lane. Completion callback leaving actual_length at zero: 7. urb_complete_locked always writing status 0: 10. usbdev_ep_may_start ignoring the in-flight URB: 6. DISCARDURB returning 0 without doing anything: 5, then the lane runs into its per-stage alarm. A reap that skips the disconnect drain: 2. A readiness map set unconditionally: 1 -- the disconnect that must report POLLERR|POLLHUP alone reports 0x11c, because the wake byte keeps the pipe readable for good. ZERO_PACKET predicate wired to false: 2. SHORT_NOT_OK ignored: 1. Transferred-count clamp removed: 1 (actual_length 999 for a 32-byte buffer). A refused start routed through the syscall map: 1 -- that one has to be a queued follower, because a leader's refusal is a syscall return value and kIOReturnNotOpen is ENODEV in both maps, so only a follower reaches the one row where they differ (kIOReturnAborted, ECONNRESET in urb->status against the EINTR the syscall map answers and the kernel never writes into a URB).
  • Two of those ten were re-applied to the tree in its present shape and rebuilt through make USB_LOOPBACK_FIXTURE=1, to confirm the lane still kills a broken engine now that the fixture is behind a variable: urb_complete_locked always writing status 0 gives 34 passed, 10 failed; the readiness map set unconditionally gives 43 passed, 1 failed, on revents=0x11c where POLLERR|POLLHUP alone is wanted. Both exit 1.

On the attached ESP32-S3 at 303a:1001, re-measured for this description against a clean rebuild of the tip: 63 consecutive runs of the review driver, and one run each of the other six.

GET_CAPABILITIES                                 0x11, in all 63 runs
control URB through poll+reap    revents 0x4 (POLLOUT), status 0, actual 18
short IN                                         actual 18 (not 255),
                                                 tail still 0xaa,
                                                 SHORT_NOT_OK status -121
bulk IN discard reaps ENOENT                     20 of 20 in each of 63 runs
                                                 (1260 discards, no exception)
REAPURBNDELAY straight afterwards finds it       20 of 20 in each of 63 runs
300 URBs queued on one fd                        300 queued (first error 0),
                                                 300 reaped, in each of 63 runs
200 rounds discarding 1 of 2 queued ep0 URBs     0 bystander ECONNRESET in each
                                                 of 63 runs (12600 rounds),
                                                 A settled 200/200
RESET with 3 URBs queued                         rc=0, 3 of 3, in each of 63
mixed 400-URB poll/epoll/select driver           submit 400, reap 400,
                                                 pollout 58, pollerr 0
ThreadSanitizer / AddressSanitizer+UBSan         0 reports, 7 drivers x 2
                                                 sanitizer builds, including
                                                 the 400-URB poll/epoll/select
                                                 driver whose first run once
                                                 reported the record-owner race

That last row is why the URB record's owning slot is _Atomic: it is the one field the completion callback reads before it can take any lock, and ThreadSanitizer cannot see through IODispatchCalloutFromCFMessage, so it reported the access on a cold-start run.

Summary by cubic

Implements the usbdevfs async URB path over IOKit, so SUBMITURB, REAPURB*, and DISCARDURB now transfer instead of returning ENOTTY; GET_CAPABILITIES reports 0x11. Completions and disconnect watches ride one lazy CFRunLoop thread, and each endpoint keeps at most one IOKit transfer in flight to preserve Linux's single-URB DISCARDURB, so queued transfers don't pipeline.

  • Applies Linux's URB validation order, shared 16 MB usbfs accounting, SHORT_NOT_OK, ZERO_PACKET, and disconnect-drain semantics; the old 256-URB-per-fd limit is gone.
  • Propagates disconnects to every open usbfs fd; a generated table pins each ioctl's departed-device answer and fixed wrong ones on claim/release, discard, the default arm, interface enumeration ops, and RESET.
  • REAPURBNDELAY issues the post-disconnect aborts and answers -EAGAIN until they land instead of blocking up to 2s.
  • Sync CONTROL now charges the shared byte allowance; per-endpoint FIFOs make submit/complete/restart O(1) under load; drain-timeout and late-callback paths hand back orphaned URBs and pin handles safely.
  • Adds an opt-in IOKit loopback fixture with scripted transfer outcomes; the default build links only a stub, and loopback tests use a separate binary so the shipped one can't become a stale fixture build.
  • Adds native URB tests, loopback completion/disconnect tests, departed-device ioctl coverage, and regression tests for epoll masks (EPOLLWRNORM, EPOLLRDNORM, EPOLLRDHUP).
  • Documents hardware-only gaps: isochronous URBs, async signals, clear-halt collateral, short bulk OUT status, and bounded DISCARDURB latency.