Don't wait in Thread.join(timeout) when draining supervised output

Merged
MA
5d ago
+106-17

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md file?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Changes To Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

What

_OutputPump.drain() no longer waits on the reader thread with Thread.join(timeout). The reader sets a threading.Event when run() ends (including when it raises), and drain() waits on that instead. The drain's idle, active and wall deadlines from #7650 are unchanged, as is POSIX behaviour.

Why

The Windows flakes in test_supervisor.py (#7819) aren't caused by pipe handles. They come from a CPython bug: on 3.13 to 3.15, Thread.join(timeout) can ignore its timeout (python/cpython#161712). ThreadHandle_join computes timeout_ns = Py_MAX(_PyDeadline_Get(deadline), 0), and Py_MAX is a macro, so it reads the clock twice. If the second read is negative, the next park gets a negative timeout, which _PySemaphore_PlatformWait treats as INFINITE. Windows hits that window far more often, because its waits end a fraction of a timer tick early and loop back around. The fix is on CPython main (gh-157495), the 3.15 backport (python/cpython#161771) is awaiting merge, and nothing is merged for 3.14. 3.11 and 3.12 use the older lock-based join and aren't affected, which is why only the 3.14 and 3.15 Windows legs failed.

drain() polled the reader with join(0.05) while the reader sat in os.read on the child's pipe. When that join lost its timeout, the drain waited until every writer closed the pipe:

  • #7807, 3.15: test_supervise_output_does_not_wait_for_lingering_descendants took 30.1 s, the lifetime of the grandchild that inherited the pipe.
  • #7803, 3.14: test_output_pump_drain_grace_ends_with_the_write (test 21 of the module, the one that never printed its .) deadlocked, because the test closes the pipe only after drain() returns.

Event.wait goes through _PyMutex_LockTimed, which reads the clock once and clamps a negative remainder to zero.

Tests

  • New: test_output_pump_drain_does_not_join_the_reader fails on the old code (drain() called join 491,037 times) and passes now.
  • Watchdog: an autouse fixture arms faulthandler.dump_traceback_later(120, exit=True) for every test in the module. Some waits there can't be bounded from Python, e.g. a pipe read held open by a stray process, Popen.__exit__'s untimed wait(), or a stuck interpreter call like this one. So a regression now fails in minutes with every thread's stack instead of using up the job. The dump goes to the real stderr, taken through capfd.disabled(); pytest's fd capture would otherwise swallow it when the process exits.
  • Unchanged: the existing tests and their bounds, including elapsed < 10.

Verification (Windows 11, ARM64)

  • Reproduced the real hang: about 1 in 30,000 runs of test 21's exact steps on native CPython 3.14.2. faulthandler showed the main thread stuck for over 20 s in Thread.join(0.05) inside drain(), with the reader in os.read.

  • Forced the bug deterministically by making Thread.join ignore its timeout:

    ScenarioOld codeNew code
    Test 21 drain-gracedeadlock (cut off by a 25 s watchdog)drain returns in 0.19 s
    Lingering grandchild, sleeper shortened to 15 ssupervise_output returns after 15.32 sreturns after 0.70 s
  • Stress run: 18,000 iterations of test 21 on the fixed code, no hang.

  • Module: test_supervisor.py passes on x64 3.14.10 and native ARM64 3.14.2 (16 passed, 6 POSIX-only skips).

  • Lint and types: ruff is clean. pyright 1.1.411 reports the same 3 errors as unmodified main, all signal.SIGKILL references in Windows-skipped tests.

  • Not run locally: Linux and macOS.

Not changed

Other timed joins exist (reflex_base/event/processor/blocking.py, reflex-bench). They wait on threads that finish promptly, so the CPython bug can only stretch a short wait there.

Closes #7819

šŸ¤– Generated with Claude Code