Prevent WebGL atlas pages from exceeding texture capacity

Merged
AN
4mo ago
+514-3

Part of: https://github.com/xtermjs/xterm.js/issues/6038
Part of: https://github.com/microsoft/vscode/issues/322756
Follow-up to: https://github.com/xtermjs/xterm.js/pull/6042

  • Keep the existing exact-four, same-sized atlas page merge behavior.
  • Evict and rebuild the glyph atlas when it reaches maxAtlasPages and that merge is unavailable, instead of adding a page the renderer cannot use.
  • Apply the same capacity check before creating an oversized-glyph page.
  • Increment pageLayoutVersion after eviction so every renderer sharing the atlas rebuilds its cached glyph mappings.
  • Clamp the texture upload loop as a defensive backstop so an unexpected overflow cannot throw while rendering.
  • Add deterministic Chromium coverage for normal page overflow, oversized glyphs, shared terminals, rendered pixels after eviction, and mid-frame retry stability.

TDD evidence:

  • Red: 8237004d5 reproduces both overflow paths on the PR 1 baseline with TypeError: Cannot read properties of undefined (reading 'version').
  • Green: 2b982bc39 keeps the atlas within renderer capacity and passes both tests.

Steps to verify:

  • npm run build
  • npm run esbuild
  • npm run esbuild-demo-client
  • npm run esbuild-demo-server
  • npm run test-integration-chromium -- --suite=addon-webgl
  • npm run lint-changes

Inspirations from:

  • Preserving exact-four merging comes from the guard added to fix the earlier stack overflow in TextureAtlas._createNewPage.
  • The capacity invariant comes from GlyphRenderer allocating its shader and texture arrays from TextureAtlas.maxAtlasPages, including one texture per supported page in the _atlasTextures initialization.
  • The eviction cache reset follows the existing cache-clearing behavior in TextureAtlas.clearTexture, while additionally removing the discarded pages.
  • Handling oversized glyphs at the same capacity boundary comes from the existing direct _overflowSizePage insertion path in TextureAtlas.
  • Mid-frame eviction recovery relies on the existing full-model retry behavior in WebglRenderer.renderRows.