Flush GZip output for each streamed chunk

Merged
KL
3mo ago
+54-8

Compress and emit output for every streamed chunk (Z_SYNC_FLUSH), instead of letting zlib buffer across chunks. Chunk boundaries are the application's intent - a compressor should not erase them.

This deliberately reverses the conclusion of #2753 and supersedes #3230 (see also discussion #3231). The precedent moved since #2753 was closed: Django now flushes every streamed chunk unconditionally (12bb16da8f, "Fixed #36293 -- Avoided buffering streaming responses in GZipMiddleware"), and Tomcat, Undertow, every Netty-based stack, Apache mod_deflate, h2o, Deno's runtime, and salvo all do the same, with no configuration knob.

  • Non-streaming responses are unaffected (single-shot bodies were already finalized immediately).
  • text/event-stream remains excluded by default. With this change, compressed and timely SSE is one setting away: exclude_content_types=().
  • The worker-thread path (#3410) is unchanged - the flush happens inside _compress_body on either path.
  • Cost: compression ratio drops for streams that emit many small chunks (~5 bytes overhead per flush plus reduced entropy pooling). Applications wanting maximum ratio can batch their chunks.

Stacked on #3418.

AI Disclaimer

This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.