Answer early scan-out requests for the nested size and row-padded modes

Merged
UT
13d ago
+193-11

Cause

At session start the first window-sized scan-out allocations reach the daemon as length-only requests and are refused with EINVAL, so they fall through to the kernel and become foreign buffers (copied by #28, dropped before it). It is not a "no mode yet" problem: the daemon notes the panel mode when the presenter is created, before any request. What it does not know yet is:

  • gamescope's nested size (-w/-h, the container screen size: 1280x800 on the Tab S7, 1280x720 on the OnePlus 12R). Nothing tells the shim or the daemon about it; the length index learns it only from the first ADDFB2 of that size, and that framebuffer is one of the buffers that were already allocated from the kernel. That is the "about 1 s later the same request succeeds" pattern.
  • the real length of a row-padded mode buffer. The panel is noted with its tight w*h*4 length; on the OnePlus 12R (2780x1264) gamescope asks for 14237696 B (rows padded to 11264 B), which only resolves once the spare or a framebuffer registers its st_size.

The shim refusal is final on attempt 1 (elapsed=0ms): the daemon answers at once, so this is not a timing or retry issue.

Change

  • gndrm-shim: every shim (gamescope, Xwayland, the guest) seeds GNDRM_SCANOUT_NESTED next to GNDRM_SCANOUT as a known mode, so window-sized allocations go out as geometry requests from the first one. The app is expected to export the variable from the same values it passes gamescope (GameNative #2267). A pre-set value always wins: the constructor returns early when the variable exists, and it uses setenv(..., 0). Only when it is absent does the shim in the gamescope process fall back to reading gamescope's own -w/-h (--nested-width/--nested-height, up to --) and export it for the children.
  • gndrm-daemon ReusePool::resolve_len: when no registered length matches, a known geometry whose linear length with rows padded to a power-of-two alignment (64..4096 B) and page rounding is exactly the request serves it, least padding first. A registered length still wins. The resolved geometry is only a guess at the caller's layout: two geometries can share a length (tight 1280x960 and 1280x800 with 6144 B rows are both 4,915,200 B), so every hand-out of a length-only request (reused buffer, spare or fresh allocation) is checked against the buffer's fstat size through scanout::serves_len, and a buffer shorter than the request is refused with EINVAL, the same answer as an unknown length. That check already existed on all three paths; it now goes through one host-tested helper, and the resolve_len doc says the caller must make it. Five unit tests, including the 1280x960 / padded 1280x800 collision.
  • gndrm-shim: the scan-out exchange resumes sendmsg/recvmsg on EINTR like the RPC path does. The verification run showed one request lost to a signal (allocation failure ... (errno 4 Interrupted system call)), which also left the daemon's reply queued on the channel.

Zero-copy and the commit path are unchanged; nothing waits.

Tab S7 (SM-T875, Android 11, ION), Legacy testbed APK, TestD3D cube + WrapperTest gl/ (hold 60 s), GNDRM_LOG_SHIM=1

Before (shim from master, route27 run today):

scanout: alloc len=4096000 -> length-only request
scanout: daemon allocated no 4096000 B buffer: unsupported geometry/format (elapsed=0ms attempt=1/3) (errno 22 Invalid argument)
scanout: ion alloc len=4096000 heaps=0x2000000 left to kernel -> fd ino=3926223
gndrm-daemon: foreign dma-buf ino=12478481 1280x800 fourcc=AR24 pitch=5120 -> copied into pool (reason: not pool-backed)

After (this branch):

09:12:14.155 [gndrm-shim] scanout: alloc len=4096000 -> mode-matched 1280x800
09:12:14.161 [gndrm-shim] scanout: heap alloc 4096000 B -> daemon buffer 1280x800 fd=29 ino=3694053 stride=5120
  • 17/17 4096000 B and 6/6 16384000 B requests answered -> daemon buffer; no daemon allocated no line for any frame length (one for a 106496 B allocation, which is not a frame); no foreign dma-buf line in either phase.
  • Cube: PowerMetrics 59.2-59.8 fps, exitStatus=0 durationSec=201 rendered=true maxFps=59.8 errors=0.
  • GL: VERDICT run=1 [gl/] PASS p=12 f=0 s=0 astcF=0 selftest=none(native) upload=- layer=0 98s; the GL window is visible during the hold.

Host: cargo test -p gndrm-daemon -p gndrm-android green (rust:1-bookworm); shim built with NDK 27.3 -Werror, export check passes, no libandroid. Parser checked on host: -w 1280 -h 800 -- game -w 9 -h 9 gives 1280x800, --nested-width=1366 --nested-height=768 gives 1366x768, run through the dynamic loader works, a preset value is kept, no socket leaves it unset.

The test payload was the route27 payload (20261027.1 base + #27 ICD) with this shim. On the Tab S7, the plain gamescope-gamenative-20261028.2.tzst stalls wine at explorer start for about 100 s and fails caps gl with the old shim as well, so it was not a usable base. It differs from the route27 payload only in libadrenotools_gamescope.so.