feat(scaling): fan-out debounce (#7756 lever 3 prototype)

Closed
JO
5mo ago
+236-1

Summary

The scaling-dive doc (#7765) identifies fan-out as the dominant cost at high concurrency:

  • `etherpad_changeset_apply_duration_seconds` stays under 5 ms even at 200 authors — server-side apply is not the bottleneck.
  • `etherpad_socket_emits_total{type=NEW_CHANGES}` scales O(N²): 1 160 emits/dwell at 20 authors becomes 66 032 at 200 authors.
  • Baseline cliffs at ~250 authors (errorRate flag fires).

This is the prototype for spec lever 3 — coalesce per-pad fan-out within a debounce window. With `fanoutDebounceMs: 50` (or any value > 0), rapid commits batch into a single `updatePadClients` invocation per pad per window. With `fanoutDebounceMs: 0` (the default) the legacy behaviour is preserved byte-for-byte.

Implementation

  • New setting `fanoutDebounceMs: 0` (default 0). Production deployments are not affected.
  • New module `src/node/handler/FanoutScheduler.ts` (~30 lines) holding the per-pad `Map<padId, Timeout>` and the public `scheduleFanout(padId, callback)`. Lives in its own file so it can be unit-tested without standing up the full pad/DB/socket.io stack — extracting it avoided a module-load cycle when the test imports `PadMessageHandler` directly.
  • `handleUserChanges` replaces `await exports.updatePadClients(pad)` with `scheduleFanout(pad.id, runFanout)`. Errors from the deferred fan-out route through `messageLogger` via `setErrorHandler`.
  • Initial-join catch-up at line ~1377 keeps the synchronous call — a joining client must not miss any revisions, no batching there.

Tests

`src/tests/backend-new/specs/fanout-debounce.test.ts` — 5 passing tests:

  • debounce=0 passthrough fires every call
  • debounce>0 coalesces N rapid calls into one
  • per-pad independence (one pad's window doesn't gate another)
  • after-flush rescheduling starts a fresh window
  • error routing through the configurable handler

Scoring this

The dive workflow doesn't have a `fanout-debounce` matrix entry yet — that's a small follow-up in ether/etherpad-load-test. For now, the lever can be scored manually:

```
gh workflow run "Scaling dive" --repo ether/etherpad-load-test \
--ref main \
-f core_ref=feat/fanout-debounce \
-f sweep='authors=20..200:step=20:dwell=10s:warmup=2s'
```

(Plus a sed step in the workflow to inject `fanoutDebounceMs: 50` — I'll send that as a separate ether/etherpad-load-test PR.)

Test plan

  • tsc clean
  • 5/5 fanout-debounce tests pass
  • 3/3 prom-instruments tests still pass (no regression in #7762 work)
  • CI green
  • After merge, score with the dive workflow against `develop` post-merge — expect `emits_new_changes` to drop and client p95 to improve at step ≥ 100

🤖 Generated with Claude Code