docs: scaling dive 2026-05 (closes Phase 2 of #7756)

Draft
JO
10d ago
+447-0

Summary

Numbers-backed answer to #7756: "how many editors can be on one pad, and where is the bottleneck?" Every claim links to a CI run whose report.json is downloadable for re-analysis.

This PR description is intentionally brief; the doc itself (docs/scaling-dive-2026-05.md) is the source of truth and contains the full methodology, every lever scored, the negative results, and the honest re-evaluations.

Headline findings

  1. Real ceiling on a 4-vCPU GitHub runner: ~350-400 authors per pad. Beyond that, p95 cliffs from ~80 ms to ~2000 ms within one step. CPU consumed 7+ cores of work per wall-second — OS preemption, not slow code paths.

  2. The "250-author cliff" earlier reports kept hitting was a measurement artefact. Per-IP commitRateLimiting + harness colocation (all simulated authors share 127.0.0.1). Fixed in load-test#105. Production deployments with many client IPs are not affected.

  3. Runner noise is significant. Each GHA matrix entry runs on a potentially different physical runner. The same develop baseline produced p95 between 38 ms and 122 ms at step 350 across three identical runs. All single-run lever conclusions sit inside the noise envelope below the cliff. The doc adopts N=3-per-lever methodology and reports min/median/max.

  4. Two changes hold up under N=3 scoring:

    • #7774 — engine.io socket flush deferral. Tightens the tail at mid-load (step 350 max: 122 → 110 ms; step 300 max: 71 → 58 ms). Wire-compatible, server-side only. The only N=3-confirmed perf win in this program.
    • #7768 — per-socket fan-out serialization. No measurable perf benefit (the earlier "70% drop" was single-run noise), but a real correctness fix: the previous code was racy under concurrent commits and could lose revisions on emit failure. Recommend merging for correctness, not perf.

  5. Five directions did not pan out, documented in full:

    • WebSocket-only transport — reliably worse at cliff (N=3 confirms).
    • --max-old-space-size heap bump — no effect.
    • Closed PRs: #7766, #7770, #7769, #7772. Each has a one-paragraph diagnosis in the doc.

  6. The cliff is hardware-bound, not code-bound, on the runner we measure. Production with more cores per host sees proportionally higher ceilings without code changes. Remaining application-level surface explored end-to-end.

What changed since the first draft of this doc

  • Original draft claimed lever 3 (#7768) gave a 70% p95 drop — this was a single-run outlier comparison and is corrected throughout.
  • New methodology caveat section explains why matrix-entry-on-different-runners means within-run cross-lever comparisons are cross-hardware.
  • N=3 envelope tables added for develop baseline, #7768, and #7774.
  • Lever 8 (#7772) closed with diagnosis; lever 8b (#7774) added.
  • Recommendation section reordered: flush deferral first, serialization second-for-correctness.

Test plan

This is documentation only — no code changes. Reviewers should verify:

  • The numbers in the tables match the linked CI runs' report.json files.
  • The methodology caveat section accurately describes how GHA matrix entries map to physical runners.
  • Recommendations match the actual disposition of each linked PR.

🤖 Generated with Claude Code