Scan-out layout per session: BGRA by default, RGBA where SurfaceFlinger cannot draw BGRA

Merged
UT
12d ago
+1503-372

Why

A RedMagic Astra (nubia NP05J, Android 16) user hit a soft reboot a few seconds after every gamescope launch. Its SurfaceFlinger runs Skia on GLES and aborts when it has to draw our scan-out buffer on the GPU:

Fatal signal 6 (SIGABRT) in tid (RenderEngine), pid (surfaceflinger)
Abort message: 'Unable to generate SkImage. isTextureValid:1 ... GrGLTextureInfo: fTarget: 36197 fFormat: 32856 colorType 6'

The pool is BGRA_8888. Android 16's Skia tags that buffer with kBGRA_8888 (colorType 6) while creating an RGBA8 external texture for it, a pair its GLES caps never accept. Android 14 and 15 draw the same buffer correctly (measured on a Poco F6 and an Odin 3, both Skia GLES, both composing the layer on the GPU), and a Vulkan or Graphite render engine has a native BGRA format.

What

  • The scan-out layout is chosen per session (modules/gndrm-android/src/layout.rs). BGRA_8888 stays the default, so every device that works today keeps exactly what it has, including direct scan-out of Xwayland's XR24 buffers. RGBA_8888 is chosen when the SDK is 36 or newer and the render engine is GLES (debug.renderengine.backend, else the vulkan_renderengine / graphite_renderengine aconfig flags read from the aconfig_flags.pb files), or when GNDRM_SCANOUT_LAYOUT is set in the environment. One unconditional log line states the decision and why.
  • In the RGBA layout the primary plane offers XBGR8888/ABGR8888 (+RGB565, +XBGR2101010 under HDR); gamescope's existing fallback (patch A46) renders XB24, so gamescope is unchanged. The cursor plane keeps ARGB8888 and XRGB8888 with its own IN_FORMATS blob so those formats stay in the union gamescope builds its client dmabuf list from.
  • The cursor plane refuses any rectangle larger than DRM_CAP_CURSOR_WIDTH/HEIGHT, as a kernel driver does. Without it libliftoff parked a full-size game layer on the cursor plane (the presenter publishes only the primary plane) and the panel stayed black.
  • Foreign buffers whose channel order differs from the pool's are swizzled while copied into the pool.
  • The ICD shim's default swapchain format follows the layout (B8G8R8A8 under BGRA, R8G8B8A8 under RGBA); GN_VKSHIM_ALLOW_RGBA=1 / GN_VKSHIM_ALLOW_BGRA=1 allow the other one. The C shim's request format follows the same variable. A request that does not match the session layout is refused with EINVAL and logged.
  • nativeScanoutLayout (JNI) returns the decision for the launcher, which passes it to the guest.

Cost

None on devices that keep BGRA. On the RGBA path GL games through Xwayland (XR24 pixmaps) are composited by gamescope instead of scanned out directly: on the Odin 3 Celeste runs at 60 fps under BGRA and 57 fps under RGBA.

Verification

  • Host tests (Linux container): uapi, daemon (lib + golden + diag), android (layout decoder, rule table, dump order), ICD shim: all green, both layouts covered.
  • Odin 3 (Android 15, Adreno 830), payload 20261029.3 + this daemon:
    • default: scan-out layout bgra (rule: sdk=35 re=gles); Celeste 60 fps and Half-Life 2 60 fps, correct colours, game buffers AR24 scanned out directly (five 1280x720 pool buffers, SurfaceFlinger scales).
    • GNDRM_SCANOUT_LAYOUT=rgba in the container env: scan-out layout rgba (env), Celeste 57 fps, correct colours, AB24 1920x1080 output buffers.
    • earlier runs on the unconditional RGBA daemon: Half-Life 2 direct AB24 scan-out, LSFG x2 at 120 out / 60 in, HDR (XB30, PQ dataspace), mouse cursor, all correct.
  • The Astra itself is the reporter's device and has not been re-tested; on it the rule resolves to RGBA (Android 16, GLES).

Ships as gamescope payload 20261029.3 (20261029.1 plus this shim and ICD) with the daemon in the app's jniLibs.