feat: capability checks run in dependency waves
Merged
EV
11h ago+654+654+012345678901234567890123456789-80-80-01234567890123456789
Description
Capability checks now run in dependency waves, not one at a time. Each check starts as soon as all its dependencies have results and a slot is free, up to MAX_PARALLEL_CHECKS = 4 per run. Runs take about as long as the depth of the chain (sign-in, then lists, then samples), not the sum of every check.
Kept:
- "Not run" and rate-limit stops: "Not run" outcomes stay. A RATE_LIMITED result stops new starts, and checks already running finish.
- Inputs and hang guards:
prior_resultsstays, and each check keeps its own hang guard. The coordinator's wait is bounded too. - Ordering: discovery checks still run last. The returned report keeps run order.
- Callbacks:
on_resultruns on the calling thread, one call at a time, and gets the results in the order they finished. - Threads: workers come from
start_thread_future, so contextvars (the tenant id) carry over. - Draft runs: the lease now renews for the longest hang guard among checks that can run at once.
Thread safety:
- This PR runs on top of #16264, which makes every source-operations gateway thread safe and documents that as a contract.
- Connectors are not thread safe. So checks that need the connector instance (the fallback checks of sources without a gateway) run one at a time, while gateway checks run beside them.
For review:
- Progress still marks one check as RUNNING, while up to 4 may run.
- Tests that assert execution order or the rate-limit stop of the next check use a new
one_check_at_a_timefixture. - The draft-run fakes in the external-dependency tests now run after sign-in, as real checks do. Without that, cancel and supersede could not stop them.
How Has This Been Tested?
- New
test_check_waves.py, built on barriers and events with no sleeps. It covers:- checks that call the connector never overlapping;
- the parallel start;
- the cap;
- the rate-limit stop while a check is running;
- report order versus completion order;
- dependency stops in parallel;
- discovery last;
- the tenant context.
- A stored-progress test for results that arrive out of order.
- Unit suites pass at this tip.
- Capability-check external-dependency tests: 94 passed, three runs of the draft-run file.
- Capability-check integration tests against a local stack at this tip: 20 passed, 4 skipped.
Additional Options
- [Optional] Please cherry-pick this PR to the latest release version.
- [Optional] Override Linear Check
🤖 Generated with Claude Code
Summary by CodeRabbit
-
New Features
- Capability checks now run concurrently, with up to four checks active at once.
- Checks respect dependencies, ensuring prerequisites complete first.
- Connector-based checks remain safely serialized when required.
- Progress updates reflect checks as they finish.
- Discovery checks begin only after validation checks complete.
-
Bug Fixes
- Rate-limited runs stop starting new checks while allowing active checks to finish.
- Checks that have not started are skipped after throttling.
- Runs remain active long enough for unfinished checks to complete safely.