fix(python): back S3BufferedReader with a real BufferedReader
Summary
S3BufferedReader subclasses io.BufferedReader but never calls BufferedReader.__init__, so every method it does not override fails with ValueError: I/O operation on uninitialized object. That covers reading a file from wmill.load_s3_file_reader line by line (for line in reader, readline), into a buffer (readinto, hashlib.file_digest), and in text mode (csv.reader(io.TextIOWrapper(reader))).
Rather than overriding those methods one by one, the HTTP response's chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__. The standard implementation then provides the whole file API, and the hand-written read, read1, peek and _fill go away. The behaviours the existing tests pin are the standard ones and still hold: read(0) does not touch the stream, and peek/read1 read it at most once.
Reported in #12452, which overrides the missing methods individually; text mode and readline(None) still failed there.
Changes
s3_reader.py:_HttpChunkStream(RawIOBase) yields the response chunks;S3BufferedReaderwraps it withsuper().__init__and closes itself on__exit__. Reading beforewithraises aValueErrornaming the problem instead of anAttributeError.test_s3_reader.py: text-mode CSV across chunk boundaries, and line/readintoreads that stop without draining the stream. Both fail onmain; the 5 existing tests are unchanged.
Test plan
-
PYTHONPATH=. uv run --frozen --python 3.14 pytest tests/: 137 passed. - Over a real
httpxchunked stream from a local server (4 MB, 100k-row CSV in odd-sized chunks):read(),for line in reader, first line without reading the rest,readline(5),readline(None), areadintoloop,hashlib.file_digest,csv.reader(TextIOWrapper), andclosedafter thewithblock all behave as expected. Onmainonlyread()worked. - Python job on a worker (backend
quickjs,private,parquet,python, workspace storage on SeaweedFS) reading a 4.9 MB, 100k-row CSV throughwmill.load_s3_file_reader. With the publishedwmill1.875.0 onlyread()works;readline,for line in reader,csv.reader(TextIOWrapper),readintoandhashlib.file_digestall raiseI/O operation on uninitialized object. With thiss3_reader.pyinjected into the worker's package cache, all six succeed: 100,001 lines and CSV rows, 4,966,683 bytes throughreadinto, and a SHA-256 matching the uploaded object.
š¤ Generated with Claude Code
Summary by cubic
Fixes S3BufferedReader so the whole file API works instead of failing with "I/O operation on uninitialized object". The class subclassed io.BufferedReader without initializing it, so only the hand-written read, read1, and peek worked; readline, iteration, readinto, and text mode all broke.
The HTTP response chunks now feed a small RawIOBase, and S3BufferedReader passes it to BufferedReader.__init__, which provides the standard implementation and replaces the hand-written read methods. Adds tests for text-mode CSV across chunk boundaries and reads that stop without draining the stream; both fail on main.