Remove fork contention from native e2e startup and test admission

Merged
VA
2d ago
+990-32

Problem

The full e2e target missed its existing five-second scenario guards under shared bb1 load. Removing the eight-case admission cap alone did not fix it. The native startup profile found 49.05% of samples in Hyper-V TLB flushes: fuser's helper inheritance hook forced fork/COW of each growing test process.

Design

  • Preserve unrestricted admission, with separate pools of at most 32 async workers and 16 blocking workers. Admission never depends on either count.
  • Keep fuser's production request decoder/session. Use its public from_fd constructor after an SDK-owned mount handoff. Map the helper socket to stdin with Stdio and _FUSE_COMMFD=0, allowing Command's posix_spawn path without pre_exec or vendoring.
  • Retain helper/device ownership in the existing locked session state before helper completion and the FUSE handshake. Receive descriptors atomically CLOEXEC, rejecting missing, extra or truncated rights. Normal unmount retains its existing flags, EPERM-only helper fallback, errno priority and actual-device retirement check.
  • Compare byte keys using disjoint integer prefixes before the general suffix comparison. No representation, ordering, allocation or provider accounting changes.

Deletions

Remove the admission cap/divide-by-six formula, the fuser-owned nested unmount control, and redundant short-key general comparisons. Relative to merged #834: 990 added / 32 deleted, net +958; most additions verify helper descriptor custody and the former fork path. No scenarios or expectations were removed.

Verification

Signed head461a1cb2e089a02a32db6480b96c2ff19231b10d. Main50ff90cc5 was merged only after GitHub reported a conflict. The two conflicting byte-comparison files on main were identical to merged #834; resolution retains those assertions and the comparator. Other main changes merge unchanged.

  • Real production admission is polled with 9–31 blocked futures before any release. The old buffer_unordered(8) injected at that same choke point fails the identical admission assertion while all results still complete. Together with real Tokio worker-creation controls: 3 passed in 0.01s.
  • Actual helper launch at N=1/2: pthread_atfork counts exactly zero forks; restoring the former pre_exec hook produces exactly N and fails that bound. Socket round-trip properties check complete bytes, CLOEXEC and descriptor closure. Unmount properties compare full errno and ordered kernel/helper calls to an independent normal-unmount policy, with the lazy/fallback-all control. Six focused properties passed, including complete failed-helper histories and the actual premature-clear control.
  • Eight live Linux FUSE mount/cache/rebind/concurrent-callback tests passed. One additional ignored cancellation fixture fails with ESTALE116 vs expected ECANCELED125. Frozen pre-change #834 binary acyclic_fs-75bedb06f3eb921b reproduces that identical failure in 0.06s; core.rs explicitly maps cancellation to ESTALE since 44e8d74a5 and the fixture expects ECANCELED since a86d31ed7. Its assertion remains unchanged.
  • Full byte-order differential properties and exact N/2N general-comparison/fallback-byte/allocation bounds, including the exact original comparator control; unchanged native reader ordered observer traces and cursor controls: 18 focused Stream tests passed.
  • Production descriptor parser fuzz: its newly Linux-gated entry point completed512 pinned-nightly runs in0s (Linux only). The actual non-Linux entry point compiled on macOS; no Linux imports are in scope there.
  • Windows GNU Harness/Objects/Filesystem lib+tests compile check passed in17.24s on the review-fix tree; Stream Windows and WASM checks passed in 6.77s and 2.92s.
  • Full native target: 80 tests / 722 scenarios passed, 2.75s isolated (one pre-existing local-cloud test is ignored). Three concurrent full binaries: all passed in2.94/2.94/3.01s, scenario groups2.897/2.899/2.960s. Default tools72 passed in0.54s. Exact-head artifact root: e2e-runtime-review-s1ioyzhh.

Paired measurements at the previous e115 head (the production changes and their controls are unchanged by the main merge), frozen binaries with separately initialized provider/workspace state and the same debug feature set:

  • Pre-root main vs candidate, 72 default-tool cases: 0.92 -> 0.51s; user CPU 11.81 -> 9.71s; system CPU 18.71 -> 1.78s; minor faults 466,872 -> 125,047. Both completed all cases.
  • Forkless predecessor vs byte comparator, isolated 10k: 2.40 -> 2.32s, user CPU 2.03 -> 1.98s. Under three complete binaries, 10k follow phase 1.724–1.754 -> 1.613–1.646s. All before/after runs completed all 722 scenarios; full target times 3.12–3.17 -> 3.04–3.12s.
  • Reproduce the paired cached-binary gate: $gate=Get-Content -Raw Q:/lanes/gc-e2e-runtime-paired.py; Q:/lanes/bs.ps1 python3 -c $gate. Artifacts: e2e-runtime-paired-7dven1yv in the lane target directory.

Decisions

No deadlines, quotas, case admission limits, expectations or coverage changed. The existing session state lock continues to own all startup/retirement controls; no new wakeup/locking protocol was added. Canonical SDK recovery-helper discovery is also used for normal mount, replacing fuser's private environment/PATH helper selection. Helper spawn/wait failures are deliberately returned with their cause and retained ownership; fuser's private unmount helper swallowed those failures. Kernel refusal priority and actual-device retirement remain checked. The independent policy property covers these explicit rules, rather than claiming fuser's swallowed-helper-error behavior is preserved. Fuzz targets are Linux-only and excluded from Windows proof. No request/history/JSON accounting changes owned by hf-perf-harness.

Remaining small-CPU limitation

On preceding e866 (same runtime and nominal work), eight CPUs isolated passed all80 tests/722 scenarios in4.66s (scenario group4.082s). At three-process eight-CPU load,145/138/144 cases reached their unchanged deadlines (targets8.19/8.36/8.00s). Stop before four CPUs at the first failing group; no deadline, admission or coverage change. Failed/incomplete runs are not used as speedup measurements. The earlier diagnostic eight-CPU perf capture had3,062 samples and no loss: copying6.60%, libc comparison2.74%, malloc/int_malloc5.19%, plus distributed filesystem accounting; no lost-wakeup/global-lock stall was captured. The outstanding four/eight-CPU loaded deadline requirement is not resolved. Main's one ignored native cancellation fixture also retains its independently reproduced errno mismatch described above.

Review thread fixes

Device retirement is the only production transition that clears the pending mount. A helper exit does not establish detachment; failed attempts stay retryable. The session shares that observation instead of duplicating its poll. Complete bounded helper-failure histories assert exact errors, retained state and retry counts, including an actual premature-clear fault at N/2N and generated histories. No mutex/wait protocol changes; the state remains exclusively borrowed under its existing owner lock.

The descriptor fuzz target has one Linux-only parent module. Its real non-Linux entry point compiled on macOS via ssh ivar, with no Linux dependencies in scope. On bb1 the new Linux entry point built and completed512 pinned-nightly runs in0s. Fuzz remains excluded from Windows proof.

Review-fix gate at45cf7a380:6 mount properties passed,8 live mount/cache/rebind/callback tests passed1.04s,3 runtime/admission controls passed0.01s, Windows GNU Harness/Objects/Filesystem lib/tests passed17.24s. Full80 tests/722 scenarios passed2.75s isolated and2.94/2.94/3.01s at3x load. Native build3m14s and fuzz build3m excluded. Final461a changes only the import diagnosed as unused by that compiler; behavioral tests are reused. Artifact e2e-runtime-review-s1ioyzhh; Q:/lanes/gc-e2e-runtime-review-gate.py. The previously disclosed small-CPU loaded deficit is unchanged and not claimed fixed.