feat(web): bound per-subscriber delivery with gap frames, heartbeat and a contiguous client cursor

Merged
GN
6d ago
+1724-479

Summary

  • A slow phone made the host hold an unbounded queue per subscription, and the client applied every event it received, even after a hole. A phone could lose or double text and never notice. A silent half-open link also looked alive.
  • This change bounds what the host holds for each subscriber. The host tells the client when it drops a high-rate event. The host sends a heartbeat. The client applies events only when they are contiguous, and it judges the host from the frames it receives.

Related Issue

Refs #1070

Searched open and closed PRs for the same problem. No PR covers it. This is the third of several PRs for the read-model work in #1070. It builds on the epoch PR (#1089).

Type of Change

  • feat: new feature
  • fix: bug fix
  • docs: documentation only
  • style: formatting or non-functional styling changes
  • refactor: internal cleanup with no behavior change
  • perf: performance improvement
  • test: adds or updates tests
  • build: build or dependency changes
  • ci: CI/CD workflow changes
  • chore: maintenance or tooling
  • revert: reverts a previous change

What Changed

  • Each session subscription now feeds its connection through a bounded channel of 256 items. The old unbounded channel and the lossy ring are removed.
  • A high-rate event that finds the channel full is dropped. The host then queues one gap frame for that session ahead of the next event. The client installs a snapshot when it gets a gap.
  • A reliable event that finds the channel full closes the connection. The client resumes by cursor. The existing queue tolerance still applies behind the channel.
  • The write loop sends a heartbeat frame every 10 seconds, straight to the socket. The watchdog is unchanged.
  • The client applies an event with seq <= headSeq in order, and a later event only when it is the next seq. Any other seq is not applied. The session installs a snapshot instead, one at a time per session.
  • The client holds events that arrive before the subscribe or snapshot reply. It judges them after the reply names the replay boundary. A pending permission request is still delivered after a snapshot.
  • The client judges host liveness from any received frame. After 25 seconds of silence it probes the host, then reconnects. A session whose resubscribe keeps failing is reported as degraded alone.
  • WS_PROTOCOL_VERSION is now 4. The gap and heartbeat frames are in both protocol registries. Docs and the changelog are updated.

How It Was Tested

  • bun run ci (Biome lint + format + imports, strict mode)
  • bun run typecheck
  • bun run test
  • cargo clippy --all-targets -- -D warnings (in src-tauri)
  • cargo test (in src-tauri)
  • Manual verification completed
  • Not applicable

New Rust tests cover: a dropped high-rate event followed by one gap frame; a full channel closing the connection for reliable and idempotent events; a snapshot subscription closing the connection; heartbeat frames on a busy connection; resume by cursor after an overflow. New client tests cover: contiguity after the head, a skipped seq installing one snapshot, events that overtake the reply, the gap frame, host-frame liveness, live-only recovery, and per-session degrade.

Also ran cargo clippy --bin termul-server --features standalone-server -- -D warnings and bun run build:web.

Screenshots or Recordings

No user-visible UI change. The change affects the host and the transport only. The Rust and client tests above cover the behavior.

CI & Review Gate

  • All CI checks pass (PR Validation, Rust Checks, Build Verification, Security Scans)
  • Code review comments from CodeRabbit / Claude have been addressed or resolved
  • No unresolved review findings remain

Checklist

  • My PR title follows the conventional commit format used by this repo
  • I linked the related issue or explained why none exists
  • I updated docs when needed
  • I added or updated tests when needed
  • I verified the change does not introduce unrelated modifications
  • I read AGENTS.md and followed the contributor guidelines
  • A human reviewed the complete diff before submission

Summary by cubic

Bounds per-subscriber delivery so a slow client costs the host fixed memory and can no longer lose or double events without noticing, and a half-open link is detected from host frames instead of pings. Builds on the epoch PR for #1070.

  • Each session subscription feeds its connection through a bounded 256-item channel; a full channel drops a LOSSY event, and the forwarder yields one gap frame as soon as the channel drains (the client installs a snapshot) before closing the connection for reliable events so the client resumes by cursor.
  • The write loop sends a heartbeat frame every 10 s; the client treats any received frame as proof of life and reconnects after 25 s of silence, degrading a single failing session alone.
  • The client applies events only when contiguous (replay may have holes, live events must be next seq); a skipped seq triggers a snapshot, and a gap held during a failed sync still snaps the session once it settles (one shared subscribe channel per session, reused across retries).
  • WS_PROTOCOL_VERSION is now 4; the gap and heartbeat frames are registered in both protocol registries.

Migration — No data migration needed; clients on protocol version 3 will reconnect under the new version.

Summary by CodeRabbit

  • New Features
    • Added WebSocket heartbeat frames to help detect unresponsive connections.
    • Added gap notifications when lossy session events are dropped; affected sessions recover from a snapshot.
    • Improved event replay and sequencing so duplicate or out-of-order events are handled safely.
    • A session that cannot be resubscribed is now isolated, allowing the rest of the reconnect to continue.
  • Bug Fixes
    • Reliable-event overload now closes the affected connection, allowing events to be replayed after reconnect.
    • Updated the connection error message when a session cannot be fully resynced.