[6/6] perf(control/pg): promote dependents and job waiters over their own rows

Draft
BN
5d ago
+1334-288

Stacked on #484, merge in order. Stack: 6/6, splitting #479. This is the tip of the stack: its tree is identical to #479's head (git diff --stat between them is empty).

Summary

Completions promote the dependents they unblocked, and the concurrency waiter a completion releases, over those jobs' own rows instead of loading the whole run graph (PgBackend::load_graph + Sweep::sweep + flush). Each path runs inside the same transaction and run lock as before.

What changed

  • Dependents: each dependent settle_job_targeted left blocked with no needs outstanding is decided on its own node plus a job-scoped NeedsView (ancestors followed through matrix base ids), with the same dependency_decision semantics. Skip/error settle the dependent through settle_unrun and feed the dependents that frees into the next pass of a work queue, so a skip cascade finishes in the completion that started it, including a job that fails at promotion (unservable runs-on, denied environment, failed gate), whose dependents previously waited for a later sweep. A reusable caller takes its JobSet gates (shared with the sweep); a deferred matrix goes to the expansion queue; everything else enters promote_one (#5). New control/pg/dependents.rs; PgBackend::promote_unblocked is deleted. Sweep stays for submit, rerun, cancel, expansion and the reaper.
  • Job-level concurrency waiters: a waiter is promoted from its own rows (dispatch.rs, rerun.rs), not the run graph.
  • Tests: the shared suite pins dependent promotion on both backends (dependents of a matrix base get merged outputs and wait for every leg, a failed leg skips dependents transitively, if: always() after a failure, failure() reading an ancestor reached only through a matrix base, a reusable caller that depends on another job). A scenario for a job that fails at promotion skipping the jobs behind it in the same completion fails on the sweep-based promotion (that commit is red by design until the "promote the dependents a completion unblocked over their own rows" commit that follows it in this PR) and passes after. PG tests for waiter promotion from rows.
  • Docs: internal/settle-redesign.md and CHANGELOG describe completions as graph-free (NeedsView, the dependents work queue, skip cascade), correct the ancestor query to follow matched rows through matrix base ids, and drop the graph-backed caveat.

Measurements

Lab harness on a laptop (benchmarks/control-load profiles, 2 nodes, Postgres 18), 200-leg workloads. Changed arms are 3 clean rounds; baselines are 1 clean round (main and #378 + review fixes). Treat these as direction, not a capacity claim. The "This stack" arm is the full #479 head, i.e. this tip.

WorkloadMetric#378 + review fixesThis stack
200 legs in one job-level concurrency: groupgraph loads / completion0.9950
settle lock hold, mean23.9 ms (main: 28.3)14.8 ms [12.8-16.0]
200 legs + always-run mergemerge-promoting completion~35 ms of graph work~2-6 ms of row work

(The capped max-parallel: 20 numbers are in #5.) On the plain 200-leg workload, leg completions run the same statements as before. Mean hold over 3 rounds was 16.5 ms [14.6-19.4] vs 14.8 ms [13.4-16.2] for the baseline; ranges overlap and the difference tracks run-row lock contention, not added statements. Claims still load the graph on the runner claim path.

Not included

  • Runner claim refactor, the remaining_needs decrement counter (it slowed completions under overlapping runs and is held back), and the post-completion reply read.
  • Workflow-level concurrency resume (resume_held_run), submit, rerun, and expansion (claim_expansion/apply_expansion) still load the graph.
  • The lock-scope redesign: a lock-order audit found it needs a larger design and exposes 12 pre-existing ordering problems; not attempted here.

Test plan

On this PR's tip, against a disposable Postgres 17 (max_connections=2000, PRELOOP_PG_WRITERS=3 PRELOOP_PG_READERS=3, PRELOOP_TEST_POSTGRES_URL set):

  • cargo fmt --all --check: clean
  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo test --locked -p preloop-runner-server --lib -- control::: 367 passed, 0 failed, 0 skipping: lines (same count as #479's head)
  • just test-ci (fmt-check, clippy, zizmor, PROPTEST_CASES=8 cargo test --locked --workspace, control migrations, just conform): passed; the workspace run was 2811 passed, 0 failed, 3 ignored over 50 test binaries, 0 skipping: lines, conform: PASS.
  • git diff --stat split/6 pr/graph-free-scheduler: empty (identical trees, 65a67a12fb90f7b2b8ab85091738b472d093bcb2).

The local engine CI run (ci.yml) was not run: the laptop engine's runner pool does not advertise the preloop-cpane label it requires, so jobs fail at enqueue. The checks on this PR are the first run of ci.yml for these commits; the earlier PRs in the stack (#480-#484) were verified with the control suite, fmt and clippy only, not just test-ci.

Stack

  1. #480 fix(pg): max-parallel stall, cohort waiter retry, nested-caller fold, non-terminal guard
  2. #481 fix(pg): deferred reusable callers
  3. #482 refactor(pg): remove the in-memory completion path
  4. #483 refactor(pg): row-based enqueue target, needs view, settle hand-off (no behaviour change)
  5. #484 perf(control/pg): max-parallel admission over rows
  6. #485 perf(control/pg): dependents and job waiters over rows

Each PR's base is the previous PR's branch, so each shows only its own diff. Supersedes #479 (closed).