[5/6] perf(control/pg): admit max-parallel legs over their own rows
Stacked on #483, merge in order. Stack: 5/6, splitting #479.
Summary
A completion that frees a max-parallel slot now admits the next parked leg over that leg's own rows instead of running the whole-run promotion sweep. This replaces the whole-graph trigger #1 added for the stall fix (cohort_has_admissible_leg is deleted), so a capped matrix no longer loads the run graph on each completion while a parked leg and a free slot coexist.
What changed
- New
control/pg/promote.rs:promote_oneruns the steps after a job's needs settle (cohort slot, needs andruns-onhydration, label check, environment approval, concurrency gate, enqueue) from the job's ownjobs/job_specs/job_messages/job_needsrows, its direct needs (NeedsView, from #4) and the run's submission payload, instead of a loadedRunGraph.settle_unrunsettles a job that never ran (unservable labels, denied environment, failed or cancelled gate) and refreshes its dependents. admit_cohortsadmits the parked legs of the cohorts a completion freed a slot in, in job-id order, one candidate query per step. A leg that is skipped or fails at itsif:or at a gate frees its slot and the loop continues.- The environment name, environment verdict and job gate evaluation are shared with
Sweep::promote(viapromote.rs) so the two paths cannot drift.graph.rsgainsload_node/load_run_scopeto build a node from its own rows. - Test:
max_parallel_admission_continues_past_a_leg_that_fails_its_gate. - Docs:
internal/settle-redesign.mdand CHANGELOG replace the "capped cohorts still load the run" caveat with the job-scoped admission and narrow the remaining graph load to dependent promotion.
Measurements
Lab harness on a laptop (benchmarks/control-load profiles, 2 nodes, Postgres 18), 200-leg workload, max-parallel: 20, always-run merge. Changed arm is 3 clean rounds; the baseline (#1's fixes on top of #378) is 1 clean round. Treat as direction, not a capacity claim.
| Metric | #378 + review fixes | After this stack |
|---|---|---|
| graph loads / completion | 0.900 | 0.005 |
| settle lock hold, mean | 132.7 ms | 68.2 ms [62.3-72.4] |
lock_run wait | 328 ms | 105 ms |
These numbers were taken on the full #479 head, which is the tree of #6's tip. I did not measure #5 in isolation: here the capped legs are admitted over rows, but dependents the completion unblocks (the always-run merge) are still promoted over the loaded graph until #6. The original #378 stalls on this workload (60 of 603 jobs finished); main could not run the always-merge profile. These arms still load the graph on the runner claim path (871 loads in the capped arm); the claim refactor is not part of this stack.
Not included
Dependent promotion over rows and job-waiter promotion (#6). The runner claim refactor, workflow-level concurrency resume, submit, rerun and expansion still load the graph.
Test plan
On this PR's tip, against a disposable Postgres 17 (max_connections=2000, PRELOOP_PG_WRITERS=3 PRELOOP_PG_READERS=3, PRELOOP_TEST_POSTGRES_URL set):
cargo fmt --all --check: cleancargo clippy --workspace --all-targets -- -D warnings: cleancargo test --locked -p preloop-runner-server --lib -- control::: 352 passed, 0 failed, 0skipping:lines
just test-ci and the local engine CI run were not run on this PR's tip: the laptop engine's runner pool does not advertise the preloop-cpane label that ci.yml requires, so jobs fail at enqueue. The checks on this PR are the first full CI run for these commits.
Stack
- #480
fix(pg): max-parallel stall, cohort waiter retry, nested-caller fold, non-terminal guard - #481
fix(pg): deferred reusable callers - #482
refactor(pg): remove the in-memory completion path - #483
refactor(pg): row-based enqueue target, needs view, settle hand-off (no behaviour change) - #484
perf(control/pg): max-parallel admission over rows - #485
perf(control/pg): dependents and job waiters over rows
Each PR's base is the previous PR's branch, so each shows only its own diff. Supersedes #479 (closed).