perf: compile-check reads the project's directories and build records in parallel

Merged
HA
1d ago
+677-97

Summary

  • uloop compile-check reads the project's directories with several readers at once, and reads the response files and the last build's log while it does, so planning waits less on every run of a large project.
  • Which assemblies it selects, which sources it compiles, and which errors it returns, in which order, do not change.

User Impact

  • Before: on a project with 561 assemblies and about 75,000 listed entries, listing the project took about 350 ms, and reading the response files and the build log added about 156 ms after it, all one read at a time.
  • After: eight readers list the directories, and the response files and the build log are read at the same time as the listing. Measurements on a large project are pending.

Changes

  • The walk now reads every directory it would descend into ahead of time with eight readers, then puts each root together from those listings in the order filepath.WalkDir would have met them: names in name order, each directory walked to the end before the next name. Only the reads run in any order; the order that decides which of two same-named .asmdef files wins and which error is reported is the same as before.
  • The walk still stops at the first error of a root, drops that root's directories, and leaves later roots unwalked. A listing whose read failed is never walked, even if the read returned part of the directory, and nothing after it in that root is picked up. A directory the read-ahead did not read is read from the disk when the walk reaches it, so the result never depends on what was read ahead.
  • Planning reads the response files, walks the project and reads the build log at the same time, and waits for all three before it decides anything, including when the response files are missing or broken. An --all run still does not read the build log.

Intended differences

  • A Windows share root used directly as an index root (\\server\share) is now listed under the root's own key. Before, its entries went under a differently spelled key with a trailing separator. This cannot be set up in CI or on the machines available, so it has no test.
  • The helper that derived a parent's key from a child's path is gone, and with it the one test that pinned its missing-parent case. Each listing is now stored under the path of the directory that was read, so a child can no longer point at a parent listing that was never started. This is the only existing test removed; no existing test's expectations changed.

Verification

  • A differential test compares the whole listing (roots, .asmdef and .asmref files in order, walk errors as text, and every directory's entries) with the previous filepath.WalkDir walk, kept in the tests as an oracle, on: nested directories whose walk order differs from path order, ignored names, links inside a root, a directory named like an .asmdef, a root that is a link, a missing root, an unreadable directory in a later root, an unreadable root, siblings and another subtree after an unreadable directory, definitions found before one, and a local package inside Packages. It passed against the old walk before the change.
  • cli/dispatcher: golangci-lint fmt --diff, go vet ./..., golangci-lint run (0 issues), the complexity config (0 issues), go test ./..., and go test -race ./internal/compilecheck/... pass. Coverage with the CI exclusions: 94.25% (baseline 94.2).
  • cli/release-automation: go test ./... passes, including the architecture test on file length. The file-length check passes.
  • Mutation checks, each restored afterwards, all fail a test: walking a listing in map order; recording a failed read as an empty listing without its error; going on after the first error in a root (caught by the siblings and other-subtree cases); keeping a failed root's directories; walking roots after a failed one; following a root that is a link (Stat for Lstat); and returning from the concurrent read before every task finished.
  • Intended survivor: a read-ahead that reads only the roots still passes every test, because the walk reads whatever was not read ahead from the disk. That rule is what makes the result independent of the read-ahead, so this mutation only costs time. It is now pinned as a test case: the differential test runs every fixture with the production read-ahead, one that reads nothing, and one that reads only the roots, and making the walk's own read return an empty listing fails the last two.