scripts/measure: swift-test-bench.sh measures one swift test filter alone on the fleet

Merged
TE
8d ago
+273-0

A swift-test benchmark gets a fleet measurement route: scripts/measure/swift-test-bench.sh builds one Swift package's tests and runs one swift test --filter as a measurement step (class exclusive, label measure), so nothing else runs on the host while it measures, and the step takes the measure route's fallbacks (a paused or busy host gives the step to another measure host within minutes) instead of waiting on one mini.

Rule it follows: "No job waits on one reserved machine: every reservation (a label one host serves) has a fallback within minutes". Before this, a Swift benchmark could run only through package-test-lane.sh suite, a light step that shares its host with other steps, so its numbers also measured the neighbours. Packages under cmux-next-ios/ were refused there too, since suite takes only Packages/ paths.

cmux-ci run --class exclusive --script scripts/measure/swift-test-bench.sh --ref FEAT_CMUX_NEXT_SHA \
  --arg cmux-next-ios/CmuxMobileKit --arg TerminalInputThroughputTests/keystrokeThroughputAt500msRTT \
  [--arg source=PR_HEAD_SHA] [--arg restore_from=BASE_SHA --arg restore=PATH,PATH] \
  --artifact artifacts/swift-test-bench/output.txt
  • PACKAGE must be a directory under Packages/ or cmux-next-ios/ with no .., . or empty segment. FILTER is one test name (letters, digits, _ . / : -). Nothing else is passed through, so a step can't run an arbitrary command.
  • source=SHA measures another commit's tree with this reviewed script. The fleet runs a step script only when its bytes match feat-cmux-next. A pull request that branched before this script landed therefore runs at the feat-cmux-next tip with source= set to its head.
  • restore_from=SHA restore=PATH,... first checks out those files of the package from another commit (paths relative to the package, each a file there). That is the "before" half of an A/B whose harness exists only at the head.
  • CmuxMobileKit's CmuxMobileLinkFFI.xcframework (a gitignored binary target) is built first with cmux-next-ios/rust/scripts/build-xcframework.sh.
  • The step fails when the build fails, when the filter runs no test (no Swift Testing or XCTest count above zero), or when a test fails. It never retries. The test output, including the benchmark's print lines, is in the step log and in the artifact.
  • The whole script runs inside one function that bash reads before it runs, because source= replaces the checkout's files, this script included. On exit the checkout goes back to the step's own commit.

The controller side, which lists the script and pins the same argument grammar, env (CMUX_CI_XCODE_APP only) and artifact, is a separate private build-fleet change.

Validation

bash scripts/measure/tests/swift-test-bench-args.test.sh checks 6 accepted and 23 refused argument lists; it builds nothing. shellcheck -S warning is clean. The script's first fleet run is the A/B it was written for, and its numbers go on that PR.

Performance

No perf numbers: this adds the measurement route itself, not a speed change; its first numbers (cmux#27323's A/B) go on that PR.

Risk

This adds a new measurement script and changes nothing that already runs. It builds and runs the measured tree's own code on a measure host, like the other build-and-run measurements (tab-switch, perf-bench), under the same containment: it runs alone, under the host lock.

Agent session: agent session

šŸ¤– Generated with Claude Code