fix(testcontainers): stop a cancelled performance run from leaking its compose stack
Why
A cancelled performance run leaves its Docker Compose stack running on the CI
runner. This run
is the case that prompted the fix: the cancel signal arrived, send_results()
was still serializing the report when the follow-up signal landed, pytest died
mid-upload, and infrahub-test-b52f4d8e was never torn down.
The mechanism is a hook-ordering problem. On an interrupted session pytest does
not tear fixtures down from the test loop — class and session fixtures are
finalized by _pytest.runner's pytest_sessionfinish. Pluggy runs entry-point
plugins before core ones, so result reporting was sitting in front of the
fixture that stops the stack, and a GitHub runner gives a cancelled step only
about 7.5s between signals.
Goal: a cancel can no longer cost the stack its shutdown.
Non-goals: the workflow-side if: always() sweep of leftover infrahub-test-*
projects, which belongs in infrahub-private-tests. That is the only cleanup
that survives SIGKILL and is still worth adding — these commits shrink the
window, they do not remove the need for a backstop.
What changed
Three commits, each independently reviewable.
fix: report performance results after the stack teardown — marks
pytest_sessionfinish trylast so reporting runs behind the teardown.
fix: keep result reporting from aborting the session — finalize()
guarded only the HTTP call, and only against Exception; the serialization
ahead of it was unguarded, so a KeyboardInterrupt there escaped
wrap_session's finally and took the process down before pytest finished
unwinding. Now the whole reporting path is guarded against BaseException, and
an interrupted session is not reported at all — those numbers cover part of the
suite, and the remaining seconds belong to the teardown.
perf: encode the performance payload once per upload — the payload embeds
every metric series scraped over the run, so it grows with run length and
dominates the session-finish path. It was encoded twice (once for the checksum,
again by httpx for the body). Now encoded once, with the envelope assembled
around those bytes, plus an explicit upload timeout instead of httpx's 5s
default.
What stayed the same: the checksum contract. It still covers a compact encoding
of the payload value, so a receiver that re-encodes the parsed payload verifies
exactly as before — the difference is that those bytes are now also the bytes on
the wire. Envelope key order changes (data moves last); no other API change.
How to review
Start with plugin.py — the trylast marker and the INTERRUPTED early return
are the whole behavioral fix. _serialize_request() in performance_test.py is
the part worth extra scrutiny: it splices pre-encoded bytes into the envelope
rather than handing httpx a dict, and test_request_checksum_covers_the_payload_on_the_wire
reproduces the receiver's verification to pin that down.
test_reporting_runs_after_the_stack_teardown asserts the hook order directly,
so a future plugin change that reintroduces the bug fails a test rather than a
production run.
How to test
cd python_testcontainers && uv sync --all-groups
uv run pytest --rootdir=. -c pyproject.toml -q tests
The cancellation itself was reproduced out of tree by driving a real pytest
session with a class-scoped fixture and a slow send_results, sending SIGINT
and then a second SIGINT 3s later:
| exit | teardown ran? | |
|---|---|---|
| before | -2 (killed mid-upload) | no — stack leaks |
| after | 2 (clean INTERRUPTED) | yes |
Impact & rollout
- Backward compatibility: checksum verification unchanged; envelope key
order changes. No change to any collected metric. - Performance: halves payload encoding on the session-finish path.
- Config/env changes: none.
- Deployment notes: safe to deploy;
infrahub-testcontainersonly, no
server-side code touched.
Checklist
- Tests added/updated
- Changelog entry — not applicable,
python_testcontainers-only changes
carry no fragment by existing convention - External docs updated — n/a
- Internal .md docs updated — n/a
- I have reviewed AI generated content