test(server): warm the outbound client before the merge-poll budget
Fixes a pre-existing flake family in the merge-poll increment, not a diff from any other PR: every case reproduces on unpatched main. Between them they account for the shard-2 red on #692 (rust-test (2), run af43a7cc), the shard-1 failure on #697 (31474986-f735-4baa-a074-891b60f248a5) and the shard-4 failure on #685 (93620639-895a-4315-a233-31896c284d14) — the other shard-2 contributor was the tailscale wizard test, fixed by #704.
Rebased onto main at 69603fbc6 (no conflicts; the unrelated init.rs reformat from #696/#704 cleared the fmt gate). Head 3e61909d: the poll_only_shared warm, then the fixture-path warm.
assertion `left == right` failed
left: 1
right: 2
Root cause
poll_fresh_merge checks its budget after a probe returns, so the first probe owns the whole window a test gives it (5 s at a 20 ms interval, in most of these). That first probe is also the process's first use of the process-wide outbound client: crate::shared_http::CLIENT, a LazyLock whose initializer builds the TLS root set — the bundled Mozilla anchors, the OS certificate store (read through a daemon under shared_http's own NATIVE_ROOTS_DEADLINE = 5 s, from 44d51b523), and any PRELOOP_GITHUB_CA_FILE / SSL_CERT_FILE bundle. On a host whose trust store answers slowly, that build alone spends the budget: the loop breaks after one probe and the test fails on the certificate store instead of on the poll schedule. It is trust-store latency, not CPU — the same tests pass under pure CPU load, while a large PRELOOP_GITHUB_CA_FILE reproduces every one of them deterministically.
Fix (test-only)
Two setup helpers now complete that LazyLock build before the caller starts a poll budget:
std::sync::LazyLock::force(&crate::shared_http::CLIENT);
poll_only_shared— the state builder the sixpoll_fresh_merge-driving tests use.WebhookFixture::with_workspace— the engine fixture the webhook-delivery poll tests use (they setfixture.state.fresh_merge_pollthemselves, so the first helper never covered them).
Each carries the reason in its doc comment. No assertion relaxed, no budget raised, no sleep added, and the 20 ms interval still exercises the retry, so the assertions keep meaning what they say: a second (or third) poll must land inside the 5 s window. Both helpers are #[cfg(test)], so the engine's own client build and poll_fresh_merge's budget accounting are untouched.
Every test this covers, with its pre-fix verdict on a slowed trust store
| test | budget | pre-fix, slowed root store | post-fix |
|---|---|---|---|
github::tests::poll_slot_is_released_when_the_probe_turns_fatal | 5 s | FAIL left: 1, right: 2 @ :8102 (13.6 s) | pass |
github::tests::poll_slot_is_released_when_the_budget_runs_out | 5 s | FAIL "the probe is retried within the budget" (13.4 s) | pass |
github::tests::poll_slot_is_released_once_github_offers_the_merge | 5 s | FAIL "GitHub's merge must be used once offered, got None" @ :8051 (10.8 s) | pass |
github::tests::mergeability_unknown_is_polled_until_the_test_merge_is_ready | 5 s | FAIL (14.6 s) | pass |
github::tests::head_race_builds_and_runs_the_engines_own_merge | 5 s | FAIL (10.3 s) | pass |
github::tests::budget_expiry_builds_and_runs_the_engines_own_merge | 5 s | FAIL (16.5 s) | pass |
github::tests::merge_state_probes_in_flight_stay_within_the_probe_capacity | 600 ms | pass, 13.2 s — the build, not the probes, is the runtime | pass |
github::tests::saturated_probe_permits_skip_the_probe_and_build_the_merge | 10 s | pass, 11.7 s (holds every probe permit: no probe, no build in its window) | pass |
github::tests::poll_slot_is_released_when_the_waiting_delivery_is_cancelled | 30 s | pass, 20.9 s | pass |
github::tests::api_failure_within_the_budget_builds_and_runs_the_engines_own_merge | 5 s | pass, 8.6 s | pass |
github::tests::a_merge_commit_from_another_head_is_never_used | 100 ms | pass, 12.2 s | pass |
The six poll_only_shared tests and the fixture-path poll tests set their schedules through one of the two warmed helpers; the remaining fixture tests inherit the warm from with_workspace.
Verification performed
Pre-fix repros on unpatched main — each with PRELOOP_GITHUB_CA_FILE pointing at a 150 MB PEM bundle (repeated self-signed certs) and cargo test -p preloop-runner-server --lib -- <test> --exact: the table above, including the two panics CI reported (:8102 and :8051).
Stability after the fix
- 120 runs, 0 failures: the four
poll_only_sharedprobe tests 10× in each of three modes — plain host, the 150 MB slowed root store, and ~3× CPU oversubscription (24 spinners on 8 cores; CI guests run at ~2.7×). - Post-merge refresh on
69603fbc6: those four tests 10× plain + 1× slowed each (44 runs, 0 failures). poll_slot_is_released_once_github_offers_the_merge(the shard-4 member): 10× plain + 10× slowed (20 runs, 0 failures).- The three fixture-path tests: 10× plain + 5× slowed each (45 runs, 0 failures).
Partition runs on the final head — all four, --locked --workspace --no-fail-fast, exit 0 each: hash:1/4 824/824 (190.7 s; carries poll_slot_is_released_when_the_budget_runs_out), hash:2/4 802/802 (184.0 s; carries the three other poll_only_shared tests), hash:3/4 834/834 (202.0 s), hash:4/4 812/812 (202.8 s; carries poll_slot_is_released_once_github_offers_the_merge, mergeability_unknown_is_polled_until_the_test_merge_is_ready and budget_expiry_builds_and_runs_the_engines_own_merge). Those are the partitions whose rust-test jobs failed on #692 (shard 2), #697 (shard 1) and #685 (shard 4).
Other gates — cargo fmt --all --check clean; cargo clippy --locked -p preloop-runner-server --all-targets -- -D warnings clean; cargo test -p preloop-runner-server --lib green on the final head (1288 passed, 0 failed, 3 ignored, 1155 s).
Not the same class
github::tests::dirty_push_sync_verifies_the_branch_head_and_reports_checks_on_the_materialized_commit (tests/concurrency.rs) also failed in one of those runs with "branch feat/x not found on GitHub after a dirty-tree push". It does not go through either warmed helper, and the one-time client build happens before reqwest starts a request's own timeout, so it cannot surface as that error. Left alone.
Protocol surface
- I confirm this change touches the runner protocol interface: NO (test-only; no wire shape, status code, or lease timing)
Required gates
- fmt-check + clippy
-D warnings+cargo test -p preloop-runner-server --libpass locally - Supply-chain isolation: no dependency/policy file touched
- Protocol/wire shapes: not applicable
- No secrets, credentials, or internal/deployment-specific paths introduced (the repro fixture lives outside the repo and is not committed)
Checklist
- Tests updated: both engine-setup helpers used by the poll family now warm the process-wide client; assertions unchanged and still strict
- Docs: not applicable (test-only)
- Changelog: not applicable (test-only)
Summary by CodeRabbit
- Tests
- Updated webhook and polling tests to initialize the shared HTTP client before timed polling begins.