feat: compile-check no longer makes dependents wait for each other when an assembly's public API changes

Merged
HA
14h ago
+1229-84

Summary

  • When an assembly's public API changes, compile-check no longer makes its dependents wait for each other before compiling. A dependent that has to compile starts as soon as the assemblies selected for their own sources have finished, assuming the other dependents it references keep Unity's public surface, and compiles again only if that assumption turns out wrong.
  • Builds on #3343, which is merged.

User Impact

Before: on a project with 561 assemblies, adding a public member to an assembly that 67 others reference compiled all 68 in 35 to 43 seconds. The 67 dependents reference each other through 354 references up to ten levels deep, and each waited for its references to finish, although in every measured run only the edited assembly changed its reference assembly. The dependents' critical path, 31 to 37 seconds, was the wall time, and more --jobs did not shorten it.

After: a dependent that has to compile starts as soon as the edited assembly finishes. Simulating the measured unit times puts the critical path at 20 to 25 seconds; the floor is the edited assembly plus the largest dependent. Results are unchanged: every result reported is the compile against the right reference assemblies, after every reference finished.

A second run over the same inputs still replays every assembly instead of compiling it twice.

Changes

  • A unit may start before every plan reference has a final state when it has to compile anyway (selected for its own sources, or a dependent one of whose finished references changed), every missing reference is a dependent rather than one selected for its own sources, and it is not blocked. The references still compiling are read from Unity's build (KeptSurfaceReferences), never from an output an earlier run left.
  • A unit's final state is decided only once every reference has one:
    • every assumption held: the early compile is its result;
    • an assumed reference changed its surface: it compiles again against this run's output (at most twice per run);
    • an assumed reference failed or was blocked: it is blocked and its result dropped.
      The errors and the surface verdict of a dropped result release and block nothing.
  • A unit whose record may still be replayed once those references finish (the predicate from #3343) waits instead, so a run over the same inputs replays it.
  • Starting and settling alternate until neither moves, because a skip decided while starting units can settle a unit that already finished.
  • A skipped dependent now counts as having kept Unity's surface explicitly. Its outputs and records are still removed as before.
  • A plan that can never finish (references waiting on each other) is an error naming the units left, instead of an empty success.
  • Every final state goes through one place that returns an error if a reference is still unfinished. A unit queued early and then blocked while it waits for a job is only taken out of the queue; the last of its references to finish offers it again, and it is decided then.
  • UnitResult gains AssumedReferences, CompiledAgainAfter, DiscardedDuration and WaitedForRecord. The VibeLog units entries carry them as assumed_kept, compiled_again_after, discarded_ms and waited_for_record, each only when it applies. A unit replayed after waiting is left out of units like every replayed unit, so waited_for_record marks a wait for a record that then did not apply.
  • The JSON output is unchanged.
  • docs/compile-check.md step 5 and docs/vibe-logs.md describe this; ADR 0016 records the decision.

Verification

  • 22 new tests: 21 scheduler tests in scheduler_optimistic_test.go, and one VibeLog test.

    • The first 20 were written before the implementation. On the base (with only the four UnitResult fields added), 17 of them failed and the three guard tests passed, as they should:
      • a dependent waits for a reference selected for its own sources;
      • a dependent with no changed reference does not start early;
      • a blocked dependent does not start early.
    • The last two were added after review:
      • a blocked unit queued early is decided only once its references finish;
      • a unit queued early is queued once when its last reference finishes while it waits for a job.
    • Every existing test passes unchanged.
  • The tests that need a dependent to start while a reference still compiles no longer rely on a 60 ms hold. The fake compile of that reference finishes only once the dependent has begun, or after 10 seconds, leaving the test's assertions to fail. Tests that check a unit does not start keep the hold, which a slow machine can only make pass.

  • Repeated runs:

    • go test -race -count=10 -run TestCompileUnits ./internal/compilecheck/ passes;
    • 24 concurrent runs of the race-built test binary with -test.cpu=1 -test.count=10 pass: 240 runs, no failure. With the hold, 2 of 240 failed.
  • Mutations, each restored afterwards:

    MutationFails
    Settle a finished unit before all its references are final13 tests, through the error of the final-state check; with that check removed too, 9 tests, including the discarded verdict and discarded errors tests
    Never mark a unit stalecompiled again (dependent and edited assembly), stale only once more, discarded verdict and errors, the record tests
    Ignore references selected for their own sourceswaits for a reference whose own sources changed, and 6 others
    Start dependents that have no changed referencedoes not start before a reference is known to have changed; skips the whole chain below an unchanged surface
    Leave assumed references out of KeptSurfaceReferencesstarts a dependent before the dependents it references finish (it read the stale output), and 2 others
    Keep the results of blocked unitsblocks a dependent that started before its reference failed; blocks one whose assumed reference was blocked
    Drop the error for a plan that cannot finishreports a plan it cannot finish
    Never wait for a recordwaits for an assumed reference when its record may still apply
    Settle only after collecting a completionsettles a dependent whose assumed reference was skipped; blocks one whose assumed reference was blocked
    Decide a blocked unit queued early while a reference still compilesdecides a blocked unit queued early only once its references finish (the final-state check reports it)
    Drop the guard against queuing a unit twicequeues a unit once when its last reference finishes while it waits for a job (D compiled twice)

    Dropping the blocked condition from the early start is not a mutation here: the blocked check when a unit is taken from the queue stops the same unit, so nothing observable changes. The guard test pins that this check works on the early path.

  • In cli/dispatcher:

    • gofmt -l is empty;
    • go vet ./... and golangci-lint run ./... (0 issues) pass;
    • go test -race passes on internal/compilecheck/... and internal/dispatcher/...;
    • statement coverage from go test -coverprofile ./... is 94.055%, against 94.066% on the base branch measured the same way. That is 0.011 points lower, within the 0.1-point tolerance. The uncovered error branch of the final-state check cannot be reached while the scheduler works correctly.
  • scripts/check-code-complexity.sh and scripts/check-file-length.sh pass.

  • Repository CI does not run on pull requests into this branch; the checks above were run locally. Wall-time measurements on a large project are not part of this PR.