[5/6] perf(control/pg): admit max-parallel legs over their own rows

BN
5d ago
+989-315

Stacked on #483, merge in order. Stack: 5/6, splitting #479.

Summary

A completion that frees a max-parallel slot now admits the next parked leg over that leg's own rows instead of running the whole-run promotion sweep. This replaces the whole-graph trigger #1 added for the stall fix (cohort_has_admissible_leg is deleted), so a capped matrix no longer loads the run graph on each completion while a parked leg and a free slot coexist.

What changed

  • New control/pg/promote.rs: promote_one runs the steps after a job's needs settle (cohort slot, needs and runs-on hydration, label check, environment approval, concurrency gate, enqueue) from the job's own jobs/job_specs/job_messages/job_needs rows, its direct needs (NeedsView, from #4) and the run's submission payload, instead of a loaded RunGraph. settle_unrun settles a job that never ran (unservable labels, denied environment, failed or cancelled gate) and refreshes its dependents.
  • admit_cohorts admits the parked legs of the cohorts a completion freed a slot in, in job-id order, one candidate query per step. A leg that is skipped or fails at its if: or at a gate frees its slot and the loop continues.
  • The environment name, environment verdict and job gate evaluation are shared with Sweep::promote (via promote.rs) so the two paths cannot drift. graph.rs gains load_node/load_run_scope to build a node from its own rows.
  • Test: max_parallel_admission_continues_past_a_leg_that_fails_its_gate.
  • Docs: internal/settle-redesign.md and CHANGELOG replace the "capped cohorts still load the run" caveat with the job-scoped admission and narrow the remaining graph load to dependent promotion.

Measurements

Lab harness on a laptop (benchmarks/control-load profiles, 2 nodes, Postgres 18), 200-leg workload, max-parallel: 20, always-run merge. Changed arm is 3 clean rounds; the baseline (#1's fixes on top of #378) is 1 clean round. Treat as direction, not a capacity claim.

Metric#378 + review fixesAfter this stack
graph loads / completion0.9000.005
settle lock hold, mean132.7 ms68.2 ms [62.3-72.4]
lock_run wait328 ms105 ms

These numbers were taken on the full #479 head, which is the tree of #6's tip. I did not measure #5 in isolation: here the capped legs are admitted over rows, but dependents the completion unblocks (the always-run merge) are still promoted over the loaded graph until #6. The original #378 stalls on this workload (60 of 603 jobs finished); main could not run the always-merge profile. These arms still load the graph on the runner claim path (871 loads in the capped arm); the claim refactor is not part of this stack.

Not included

Dependent promotion over rows and job-waiter promotion (#6). The runner claim refactor, workflow-level concurrency resume, submit, rerun and expansion still load the graph.

Test plan

On this PR's tip, against a disposable Postgres 17 (max_connections=2000, PRELOOP_PG_WRITERS=3 PRELOOP_PG_READERS=3, PRELOOP_TEST_POSTGRES_URL set):

  • cargo fmt --all --check: clean
  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo test --locked -p preloop-runner-server --lib -- control::: 352 passed, 0 failed, 0 skipping: lines

just test-ci and the local engine CI run were not run on this PR's tip: the laptop engine's runner pool does not advertise the preloop-cpane label that ci.yml requires, so jobs fail at enqueue. The checks on this PR are the first full CI run for these commits.

Stack

  1. #480 fix(pg): max-parallel stall, cohort waiter retry, nested-caller fold, non-terminal guard
  2. #481 fix(pg): deferred reusable callers
  3. #482 refactor(pg): remove the in-memory completion path
  4. #483 refactor(pg): row-based enqueue target, needs view, settle hand-off (no behaviour change)
  5. #484 perf(control/pg): max-parallel admission over rows
  6. #485 perf(control/pg): dependents and job waiters over rows

Each PR's base is the previous PR's branch, so each shows only its own diff. Supersedes #479 (closed).