PR #479 - preloopdev/preloop

Summary

Follow-up to #378. It fixes the findings left open on that PR and removes the whole-run graph load (PgBackend::load_graph + Sweep::sweep + flush) from three more Postgres scheduler paths, so each reads only the rows it needs and runs inside the same transaction and run lock as before.

#378 merged without the review fixes. On main today a max-parallel matrix stalls once its first slots are taken (60 of 603 jobs finished in the lab harness), and nested reusable callers can fail to fold.

Review fixes for #378

Max-parallel stall: a completion that frees a cohort slot now admits a parked leg. Matrix legs never name their siblings in job_needs, so the dependents-only trigger never fired.
Nested reusable callers: try_fold_reusable_caller continues past an incomplete caller instead of returning.
Cohort re-park: a waiter re-parked at the cohort cap is retried when capacity frees on another gate (only groups with no live holder).
Non-terminal status guard: complete_job/settle_job reject non-terminal statuses before any write.
Deferred reusable callers: PG now persists the expanded caller's inner_job_ids and splices matrix parents into it, as lite does. Without this a deferred (fromJSON) matrix inside a reusable workflow never folded its caller on Postgres. Main and lite were not affected before #378.
Removes the in-memory completion path (complete_node, fail_fast_siblings) that has no callers.

Graph-free paths

Matrix slots: admission is a cap/active count in SQL plus a job-scoped promote_one for the next waiting leg. A leg that is skipped or fails at a gate frees its slot and the loop continues.
Dependents: newly unblocked dependents are promoted from their own rows. needs and if: are evaluated over a job-scoped NeedsView (ancestors followed through matrix base ids). Skip/error cascades settle in the same transaction. promote_unblocked is deleted.
Job-level concurrency waiters: a waiter is promoted from its own rows. on_job_enqueued takes a row-based EnqueueTarget instead of a RunGraph.

Measurements

Lab harness on a laptop (benchmarks/control-load profiles, 2 nodes, Postgres 18), 200-leg workloads. Each changed arm is 3 clean rounds; baselines are 1 clean round (main and PR + review fixes). Treat these as direction, not a capacity claim.

Workload	Metric	PR + review fixes	This PR
200 legs, `max-parallel: 20`, always-run merge	graph loads / completion	0.900	0.005
	settle lock hold, mean	132.7 ms	68.2 ms [62.3-72.4]
	`lock_run` wait	328 ms	105 ms
200 legs in one job-level `concurrency:` group	graph loads / completion	0.995	0
	settle lock hold, mean	23.9 ms (main: 28.3)	14.8 ms [12.8-16.0]
200 legs + always-run merge	merge-promoting completion	~35 ms of graph work	~2-6 ms of row work

Notes:

Original #378 stalls on the capped workload (60 of 603 jobs finished); main could not run the always-merge profile.
On the plain 200-leg workload, leg completions run the same statements as before. Mean hold over 3 rounds was 16.5 ms [14.6-19.4] vs 14.8 ms [13.4-16.2] for the baseline; ranges overlap and the difference tracks run-row lock contention, not added statements.
These arms still load the graph on the runner claim path (871 loads in the capped arm), since the claim refactor is not part of this PR.

Not included

Runner claim refactor (claims still load the graph), the remaining_needs decrement counter, and the post-completion reply read. The counter slowed completions under overlapping runs and is held back.
Workflow-level concurrency resume (resume_held_run), submit, rerun, and expansion (claim_expansion/apply_expansion) still load the graph.
The lock-scope redesign. A lock-order audit found it needs a larger design and exposes 12 pre-existing ordering problems; not attempted here.

Test plan

cargo test --locked -p preloop-runner-server --lib -- control:: against a disposable Postgres 17 (max_connections=2000, PRELOOP_PG_WRITERS=3 PRELOOP_PG_READERS=3): 367 passed, 0 failed, no skipping: lines. The shared suite covers lite as well.
New tests: cohort admission (including a leg that fails its gate), gate-waiter retry, nested-caller fold, non-terminal completion guard, deferred reusable callers (4 scenarios, PG and lite), dependent promotion (always-run after failure, transitive skip, ancestors through a matrix base, matrix-base dependents, reusable-caller dependents), waiter promotion from rows.
just test-ci and the local engine CI run were not executed for this head: the local engine's runner pool does not advertise the preloop-cpane label that ci.yml requires, so jobs fail at enqueue. The checks on this PR are the first full CI run for these commits.

Summary by cubic

Follow-up to #378: fixes the review findings left open on that PR and removes the whole-run graph load from three more Postgres scheduler paths — matrix slot admission, dependents promotion, and job-level concurrency waiters — so each reads only the rows it involves.

Fixes from #378

A max-parallel matrix no longer stalls once its slots are taken: a completion that frees a cohort slot now admits the next parked leg in job-id order.
Nested reusable callers fold correctly even when an outer caller is incomplete.
Cohort waiters re-parked at the cap are retried when capacity frees on another gate.
complete_job/settle_job reject non-terminal statuses before any write.
Deferred reusable callers fold on Postgres: inner_job_ids are now persisted and matrix parents spliced in.
Removed the unused in-memory completion path (complete_node, fail_fast_siblings).

Graph-free completion paths

Matrix slots: admission is a cap/active count in SQL plus a job-scoped promote_one for the next waiting leg; a leg that skips or fails at a gate frees its slot.
Dependents: promoted from their own rows via a job-scoped NeedsView (ancestors followed through matrix base ids); skip/error cascades settle in the same transaction.
Job-level concurrency waiters: promoted from their own rows via a row-based EnqueueTarget instead of a RunGraph.

Lab harness (200 legs, capped matrix): graph loads per completion drop from 0.9 to 0.005, mean settle lock hold from ~133 ms to ~68 ms. Runner claim, workflow-level concurrency resume, submit, rerun, and expansion paths still load the graph.