fix(python): back S3BufferedReader with a real BufferedReader

Merged
RU
6d ago
+56-53

Summary

S3BufferedReader subclasses io.BufferedReader but never calls BufferedReader.__init__, so every method it does not override fails with ValueError: I/O operation on uninitialized object. That covers reading a file from wmill.load_s3_file_reader line by line (for line in reader, readline), into a buffer (readinto, hashlib.file_digest), and in text mode (csv.reader(io.TextIOWrapper(reader))).

Rather than overriding those methods one by one, the HTTP response's chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__. The standard implementation then provides the whole file API, and the hand-written read, read1, peek and _fill go away. The behaviours the existing tests pin are the standard ones and still hold: read(0) does not touch the stream, and peek/read1 read it at most once.

Reported in #12452, which overrides the missing methods individually; text mode and readline(None) still failed there.

Changes

  • s3_reader.py: _HttpChunkStream (RawIOBase) yields the response chunks; S3BufferedReader wraps it with super().__init__ and closes itself on __exit__. Reading before with raises a ValueError naming the problem instead of an AttributeError.
  • test_s3_reader.py: text-mode CSV across chunk boundaries, and line/readinto reads that stop without draining the stream. Both fail on main; the 5 existing tests are unchanged.

Test plan

  • PYTHONPATH=. uv run --frozen --python 3.14 pytest tests/: 137 passed.
  • Over a real httpx chunked stream from a local server (4 MB, 100k-row CSV in odd-sized chunks): read(), for line in reader, first line without reading the rest, readline(5), readline(None), a readinto loop, hashlib.file_digest, csv.reader(TextIOWrapper), and closed after the with block all behave as expected. On main only read() worked.
  • Python job on a worker (backend quickjs,private,parquet,python, workspace storage on SeaweedFS) reading a 4.9 MB, 100k-row CSV through wmill.load_s3_file_reader. With the published wmill 1.875.0 only read() works; readline, for line in reader, csv.reader(TextIOWrapper), readinto and hashlib.file_digest all raise I/O operation on uninitialized object. With this s3_reader.py injected into the worker's package cache, all six succeed: 100,001 lines and CSV rows, 4,966,683 bytes through readinto, and a SHA-256 matching the uploaded object.

šŸ¤– Generated with Claude Code


Summary by cubic

Fixes S3BufferedReader so the whole file API works instead of failing with "I/O operation on uninitialized object". The class subclassed io.BufferedReader without initializing it, so only the hand-written read, read1, and peek worked; readline, iteration, readinto, and text mode all broke.

The HTTP response chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__, which provides the standard implementation and replaces the hand-written read methods. Adds tests for text-mode CSV across chunk boundaries and reads that stop without draining the stream; both fail on main.