Feat/spf moq latency control

Draft
A-
1mo ago
+3822-205

Six commits. The first four are bug fixes to MoQ playout timing; the last two add an
adaptive-latency controller that is off by default.


Why this exists: two real defects

Both were found while trying to hold a stable sub-second latency against a draft-19
relay, and both have observable symptoms rather than being theoretical.

1. The video self-clock free-ran from a single anchor

createVideoRendererActor anchored its presentation clock once — at the first
decoded frame — and extrapolated from performance.now() forever after. Only a
forward jump greater than 1s ever resynced it.

Three consequences, all of them one-directional:

  • Publisher/consumer clock drift integrated without bound. There was no
    mechanism at all to pull the clock back onto the delivery edge.
  • A draft-19 relay replays several recent groups to every joining subscriber, so an
    anchor placed mid-join-burst was seconds behind the live edge — and was then
    held for the life of the stream. The audio renderer had the same problem from the
    other end: it anchored its schedule at the oldest frame in the jitter buffer,
    i.e. deliberately at the back of the replay.
  • Every short pause ratcheted latency up permanently.

Symptom: playback that starts behind the live edge and drifts further behind the
longer it runs. The only recovery paths were the latency controller's 5% rate nudge
(which drains a multi-second excess very slowly) or a catch-up skip to the newest
keyframe-led group, which then plays that entire GOP.

2. The sync controller was measuring the wrong quantity

syncLatency computed newest-buffered − oldest-buffered and called that "latency".
That is jitter-buffer depth, not latency — and the renderers hold media past the
jitter buffer. The video renderer alone keeps decodeAhead frames of decoded
lookahead (default 8), which at 30fps is ~267ms sitting entirely outside the
measured window.

So a buffer that was in fact on target read as "too little latency", and the
controller applied its low-side nudge — which raises latency — until enough
un-decoded backlog reappeared to make the reading look right again.

Symptom: latency that is inconsistent between sessions and gets worse over
time, with the controller actively working against the target it was given.

Notably, bufferDepthSeconds' second parameter already documented itself as wanting
a playout timestamp. The call site was passing the oldest buffered frame instead.


The fix

Join at the live edge. Anchor at (newest buffered − target latency) instead of
at the oldest buffered frame, and re-anchor there after a catch-up skip. Audio
discards the pre-anchor backlog unheard — there is no way to fast-forward audio
without pitch artifacts — and closes the decoder on a drop, so the jumped-to audio
re-anchors rather than butt-joining after a matching stretch of scheduled silence
(which would give back exactly the latency the drop just shed). Video needs nothing
new on the decode path: the backlog is still fed to the decoder in order from its
keyframe and the existing drop-late-at-presentation path discards it against the
edge-anchored clock.

Both anchors come from the renderer behaviors, which own renderer construction and
can see each subscriber's jitter-buffer stats. The knob is latency.joinAtEdge,
reachable through MoqEngineConfig.latency, and it defaults on.

Measure edge-to-playout. syncLatency now measures the delivery edge against
state.currentTime — real latency, lookahead included. Publishing state.currentTime
also moved out of setupAudioRenderer into a new trackPlayoutTime: gated on the
AudioContext, it went silent in exactly the video-only case it exists to cover, and
it is now the controller's setpoint, so a starved clock was a stopped controller.

Bounded slew. The self-clock re-reads the delivery edge on every clock read and
slews back onto it, bounded by clockSlewRate (50ms/s default) outside a
clockSlewTolerance band. The bound is what keeps the clock monotonically forward
while being pulled backwards, and it makes the join-burst race self-correcting rather
than permanent. getJoinAnchorUs is renamed getTargetClockUs to match — it is
consulted continuously now, not once.

A fourth commit fixes a related slew bug: the budget accrues with real time since the
last evaluation, and the master-clock path returned before ever recording one. An
audio track that ended mid-stream handed presentation back to the self-clock with the
entire master-clock interval banked, cashing it in as a single lurch instead of a
50ms/s correction.


The adaptive half is opt-in and off by default

adaptLatencyTarget proposes the lowest target the observed path can sustain. It is
disabled unless asked for:

  • MoqEngineConfig.adaptiveLatency.enabled — defaults to false.
  • <simple-moq-video adaptive-latency> — presence-based, exactly like mmcc's
    autoplay attribute, and explicitly modelled on it.

With it off the reactor stays inactive: no timer, no slot writes, and every reader
resolves precisely the target it resolved before this behavior existed. That is the
control arm of the A/B this is meant to make measurable.

Precedence is one exported helper, preferredTargetLatencySeconds, giving three
modes:

  1. Measure and adapt — adaptation on, no caller-set target: adaptiveTargetLatency
    fills in.
  2. Respect a caller-set target — an explicit state.targetLatency always wins,
    even with adaptation on.
  3. Respect the MSF catalog's target — both absent leaves the
    catalog → default chain byte-for-byte as it was.

Supporting observability came along with it, because the A/B between the fixed and
adaptive mechanisms is unreadable without it: state.effectiveTargetLatency (the
setpoint actually resolved — nothing exposed this before), the cost counters
catchUpSkips / framesDropped / audioUnderruns published by a new always-on
trackPlayoutHealth, TrackSubscriberContext.arrivalJitter (a decaying
arrival-offset envelope, whose spread is the jitter primitive), and MSF jitter
(§5.2.9) surviving catalog parsing.

Caveat, stated plainly: the adaptive constants are reasoned, not measured

This matters for review, so it should not be buried. The defect fixes above are
backed by observed behaviour. The adaptive controller's tuning constants are not:

  • maxWidenRatePerSecond = 0.02 s/s
  • maxNarrowRatePerSecond = 0.005 s/s (a quarter of the widen rate)
  • widenStep = 0.15s, widenMax = 1s, widenDecayPerSecond = 0.01
  • Both rates strictly below latency.clockSlewRate / 2 (0.025 by default), behind a
    Schmitt trigger sized off the finest band any inner loop acts on.

These were derived from a separation-of-timescales argument, not from measurement.
The argument is that three negative-feedback loops act on one variable — the
bang-bang rateNudge, the self-clock's continuous slew, and this behavior's moving
setpoint — and a setpoint that moves faster than the inner loops can follow is a
setpoint they never reach. The bound is validated rather than merely documented:
resolveAdaptiveLatencyConfig derives its defaults from the latency config so an
untouched adaptive config can never violate it, and throws on an explicit value
that does.

But the specific numbers inside that bound are a judgement call. floorSeconds is
likewise a flat constant, not a derivation (hang's precedent is the same shape — a
flat 100ms fallback). A reviewer should treat them as a starting point to be tuned
against real paths, and that is a large part of why the whole thing ships off by
default.

There is also a hard observability limit worth knowing: there is no transport RTT
available. spf speaks its own draft-19 stack (no PROBE — that is a moq-lite
extension) and browsers do not populate WebTransport.getStats().smoothedRtt, so
absolute one-way delay simply cannot be measured. The estimate is built from arrival
jitter spread, the publisher's declared jitter, a constant cushion, and failure
feedback.


Relationship to the autoplay work

Integrated, not redundant — this is likely the first question, so: the three
autoplay commits (23d3ef62, cd36ce9a, 0817a879) touch none of the five
files this work centres on.

File this work centres onTouched by autoplay commits?
packages/spf/src/playback/behaviors/sync-latency.tsno
packages/spf/src/playback/actors/dom/audio-renderer.tsno
packages/spf/src/playback/actors/dom/video-renderer.tsno
packages/spf/src/playback/behaviors/dom/setup-moq-renderers.tsno
packages/spf/src/media/moq/timeline.tsno

Where the two do meet, the change is purely additive:

Shared fileThis PR
packages/spf/src/playback/engines/moq/engine.ts+59 / −6
packages/spf/src/playback/engines/moq/adapter.ts+33 / −0
packages/html/src/media/simple-moq-video/index.ts+8 / −1
packages/html/src/media/simple-moq-video/tests/index.test.ts+30 / −0

subscribe-selected-tracks.ts, its tests, and moq/tests/adapter.test.ts — the rest
of the autoplay surface — are untouched entirely.

Concretely:

  • adaptive-latency sits beside autoplay in observedAttributes and copies its
    presence-based semantics deliberately, so the two attributes behave the same way.
  • The renderers handle the deferred-audio gate as a named case, not as an
    accident. A deferred-audio unlock creates a fresh subscription whose edge can land
    behind the video self-clock, which only re-anchors on forward discontinuities and
    would otherwise freeze until the new master clock caught up — so the audio join
    anchor is clamped forward to a running video clock.
  • suspendMediaWhilePaused and the audioSuspended gate both rejoin at the live
    edge, which is the join-at-edge path this PR fixes.

Tests

1449 passing, up +39 from the 1410 baseline on mmcc/moq-sandbox.

New coverage was added as siblings of the existing fixed-setpoint tests rather
than as edits to them, so every assertion pinning the untouched chain still runs
verbatim. No autoplay test file loses a line (see the table above). The 28 removed
lines across three spf test files are helper extraction and import rewrites — for
example, the currentTime fallback test moved describe blocks because publishing
currentTime moved from setupAudioRenderer to trackPlayoutTime. No assertion was
deleted or weakened.

New test files:

  • packages/spf/src/playback/behaviors/tests/adapt-latency-target.test.ts (+410)
  • packages/spf/src/playback/actors/dom/tests/video-renderer.test.ts (+305)
  • packages/spf/src/playback/actors/dom/tests/audio-renderer.test.ts (+134)

Verified locally on this branch

CommandResult
pnpm install --frozen-lockfileexit 0 — lockfile unchanged, "Already up to date"
pnpm lintexit 0 (one pre-existing warning in resolve-catalog.ts, a file this PR does not touch)
pnpm build:packagesexit 0 — 10/10 tasks
pnpm typecheckexit 0
pnpm turbo run test --filter="@videojs/spf" --env-mode=looseexit 0 — 1449 passed, 14 skipped, 95 files passed / 1 skipped, no type errors
pnpm check:workspaceexit 1 — 9 passed, 1 failed: .claude/plans only (see below)

Known CI failure, pre-existing

The workspace job (pnpm check:workspace) will fail with:

.claude/plans: must resolve to .agents/plans

This is not caused by this PR. .claude/plans/moq-engine-implementation.md is a
tracked file added in 03cdae43 and modified by the autoplay commit 23d3ef62;
check:workspace expects .claude/plans to be a generated alias of .agents/plans,
and a real tracked directory there breaks that invariant. pnpm install says the
same thing at postinstall time: warning: refusing to replace non-generated .claude/plans.

Verified pre-existing: running pnpm check:workspace on origin/mmcc/moq-sandbox
(0817a879) with none of this PR's commits applied produces the identical result —
9 passed, 1 failed, same message.

Left alone deliberately — it is the sandbox branch owner's call whether the file
moves to .agents/plans or the check gets an exemption.


Summary by cubic

Fixes MoQ playout timing to hold steady near the live edge and adds an optional adaptive-latency controller (off by default). Also hardens session/protocol handling and validates audio config to make playback safer and more predictable.

  • Bug Fixes

    • Join at the live edge: anchor at newest buffered − target instead of the oldest frame; re-anchor after catch-up skips.
    • Measure real latency: compare delivery edge to state.currentTime (not jitter-buffer depth). Video self-clock now slews toward the edge continuously with a bounded rate; prevents drift and join-burst lock-in.
    • Move playout time publishing to trackPlayoutTime so the controller stays active in video-only cases; fix slew budget carryover when audio ends.
    • Protocol hardening: truncated control messages normalize to MoqtProtocolError; request timeouts cancel request streams so late SUBSCRIBE_OK cannot resurrect routes; pending TRACK_STATUS requests tracked and failed on teardown; raise MAX_OBJECT_PROPERTIES_LENGTH to 1 MiB.
    • Catalog/codec validation: parseChannelConfig requires a safe, finite total and handles dotted layouts; toAudioDecoderConfig derives numberOfChannels from channelConfig, validates positive integer sampleRate, and rejects bad tracks.
    • Pause/resume robustness: suspendMediaWhilePaused derives its hold window from the effective target (explicit or adaptive) and ignores non-finite/negative inputs; resume paths chain behind an in-flight suspend; resolveCatalog ignores joining-fetch entries after the settle deadline and guards callbacks by attempt.
    • simple-moq-video: report 0×0 dimensions before the first frame (canvases default to 300×150); drop NaN volume writes and clamp infinities.
  • New Features

    • Optional adaptive target latency via adaptLatencyTarget with precedence: explicit state.targetLatency > state.adaptiveTargetLatency > catalog → default. Disabled unless MoqEngineConfig.adaptiveLatency.enabled or simple-moq-video sets adaptive-latency.
    • New observability: state.effectiveTargetLatency, catchUpSkips, framesDropped, audioUnderruns, TrackSubscriberContext.arrivalJitter, and catalog jitter parsed and exposed.
    • Renderer setup passes edge anchors from subscriber stats; audio discards pre-anchor backlog, video decodes backlog and drops late; deferred-audio unlock clamps forward to a running video clock.
    • New helpers: resolveTargetLatencySeconds and preferredTargetLatencySeconds unify target resolution across consumer input, adaptive proposal, and catalog.
    • HTML element: simple-moq-video accepts adaptive-latency attribute, routed to the engine without altering numeric target-latency.