feat: capability checks run in dependency waves

Merged
EV
11h ago
+654-80

Description

Capability checks now run in dependency waves, not one at a time. Each check starts as soon as all its dependencies have results and a slot is free, up to MAX_PARALLEL_CHECKS = 4 per run. Runs take about as long as the depth of the chain (sign-in, then lists, then samples), not the sum of every check.

Kept:

  • "Not run" and rate-limit stops: "Not run" outcomes stay. A RATE_LIMITED result stops new starts, and checks already running finish.
  • Inputs and hang guards: prior_results stays, and each check keeps its own hang guard. The coordinator's wait is bounded too.
  • Ordering: discovery checks still run last. The returned report keeps run order.
  • Callbacks: on_result runs on the calling thread, one call at a time, and gets the results in the order they finished.
  • Threads: workers come from start_thread_future, so contextvars (the tenant id) carry over.
  • Draft runs: the lease now renews for the longest hang guard among checks that can run at once.

Thread safety:

  • This PR runs on top of #16264, which makes every source-operations gateway thread safe and documents that as a contract.
  • Connectors are not thread safe. So checks that need the connector instance (the fallback checks of sources without a gateway) run one at a time, while gateway checks run beside them.

For review:

  • Progress still marks one check as RUNNING, while up to 4 may run.
  • Tests that assert execution order or the rate-limit stop of the next check use a new one_check_at_a_time fixture.
  • The draft-run fakes in the external-dependency tests now run after sign-in, as real checks do. Without that, cancel and supersede could not stop them.

How Has This Been Tested?

  • New test_check_waves.py, built on barriers and events with no sleeps. It covers:
    • checks that call the connector never overlapping;
    • the parallel start;
    • the cap;
    • the rate-limit stop while a check is running;
    • report order versus completion order;
    • dependency stops in parallel;
    • discovery last;
    • the tenant context.
  • A stored-progress test for results that arrive out of order.
  • Unit suites pass at this tip.
  • Capability-check external-dependency tests: 94 passed, three runs of the draft-run file.
  • Capability-check integration tests against a local stack at this tip: 20 passed, 4 skipped.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Capability checks now run concurrently, with up to four checks active at once.
    • Checks respect dependencies, ensuring prerequisites complete first.
    • Connector-based checks remain safely serialized when required.
    • Progress updates reflect checks as they finish.
    • Discovery checks begin only after validation checks complete.
  • Bug Fixes

    • Rate-limited runs stop starting new checks while allowing active checks to finish.
    • Checks that have not started are skipped after throttling.
    • Runs remain active long enough for unfinished checks to complete safely.