Feat/spf moq latency control
Six commits. The first four are bug fixes to MoQ playout timing; the last two add an
adaptive-latency controller that is off by default.
Why this exists: two real defects
Both were found while trying to hold a stable sub-second latency against a draft-19
relay, and both have observable symptoms rather than being theoretical.
1. The video self-clock free-ran from a single anchor
createVideoRendererActor anchored its presentation clock once — at the first
decoded frame — and extrapolated from performance.now() forever after. Only a
forward jump greater than 1s ever resynced it.
Three consequences, all of them one-directional:
- Publisher/consumer clock drift integrated without bound. There was no
mechanism at all to pull the clock back onto the delivery edge. - A draft-19 relay replays several recent groups to every joining subscriber, so an
anchor placed mid-join-burst was seconds behind the live edge — and was then
held for the life of the stream. The audio renderer had the same problem from the
other end: it anchored its schedule at the oldest frame in the jitter buffer,
i.e. deliberately at the back of the replay. - Every short pause ratcheted latency up permanently.
Symptom: playback that starts behind the live edge and drifts further behind the
longer it runs. The only recovery paths were the latency controller's 5% rate nudge
(which drains a multi-second excess very slowly) or a catch-up skip to the newest
keyframe-led group, which then plays that entire GOP.
2. The sync controller was measuring the wrong quantity
syncLatency computed newest-buffered − oldest-buffered and called that "latency".
That is jitter-buffer depth, not latency — and the renderers hold media past the
jitter buffer. The video renderer alone keeps decodeAhead frames of decoded
lookahead (default 8), which at 30fps is ~267ms sitting entirely outside the
measured window.
So a buffer that was in fact on target read as "too little latency", and the
controller applied its low-side nudge — which raises latency — until enough
un-decoded backlog reappeared to make the reading look right again.
Symptom: latency that is inconsistent between sessions and gets worse over
time, with the controller actively working against the target it was given.
Notably, bufferDepthSeconds' second parameter already documented itself as wanting
a playout timestamp. The call site was passing the oldest buffered frame instead.
The fix
Join at the live edge. Anchor at (newest buffered − target latency) instead of
at the oldest buffered frame, and re-anchor there after a catch-up skip. Audio
discards the pre-anchor backlog unheard — there is no way to fast-forward audio
without pitch artifacts — and closes the decoder on a drop, so the jumped-to audio
re-anchors rather than butt-joining after a matching stretch of scheduled silence
(which would give back exactly the latency the drop just shed). Video needs nothing
new on the decode path: the backlog is still fed to the decoder in order from its
keyframe and the existing drop-late-at-presentation path discards it against the
edge-anchored clock.
Both anchors come from the renderer behaviors, which own renderer construction and
can see each subscriber's jitter-buffer stats. The knob is latency.joinAtEdge,
reachable through MoqEngineConfig.latency, and it defaults on.
Measure edge-to-playout. syncLatency now measures the delivery edge against
state.currentTime — real latency, lookahead included. Publishing state.currentTime
also moved out of setupAudioRenderer into a new trackPlayoutTime: gated on the
AudioContext, it went silent in exactly the video-only case it exists to cover, and
it is now the controller's setpoint, so a starved clock was a stopped controller.
Bounded slew. The self-clock re-reads the delivery edge on every clock read and
slews back onto it, bounded by clockSlewRate (50ms/s default) outside a
clockSlewTolerance band. The bound is what keeps the clock monotonically forward
while being pulled backwards, and it makes the join-burst race self-correcting rather
than permanent. getJoinAnchorUs is renamed getTargetClockUs to match — it is
consulted continuously now, not once.
A fourth commit fixes a related slew bug: the budget accrues with real time since the
last evaluation, and the master-clock path returned before ever recording one. An
audio track that ended mid-stream handed presentation back to the self-clock with the
entire master-clock interval banked, cashing it in as a single lurch instead of a
50ms/s correction.
The adaptive half is opt-in and off by default
adaptLatencyTarget proposes the lowest target the observed path can sustain. It is
disabled unless asked for:
MoqEngineConfig.adaptiveLatency.enabled— defaults tofalse.<simple-moq-video adaptive-latency>— presence-based, exactly like mmcc's
autoplayattribute, and explicitly modelled on it.
With it off the reactor stays inactive: no timer, no slot writes, and every reader
resolves precisely the target it resolved before this behavior existed. That is the
control arm of the A/B this is meant to make measurable.
Precedence is one exported helper, preferredTargetLatencySeconds, giving three
modes:
- Measure and adapt — adaptation on, no caller-set target:
adaptiveTargetLatency
fills in. - Respect a caller-set target — an explicit
state.targetLatencyalways wins,
even with adaptation on. - Respect the MSF catalog's target — both absent leaves the
catalog → default chain byte-for-byte as it was.
Supporting observability came along with it, because the A/B between the fixed and
adaptive mechanisms is unreadable without it: state.effectiveTargetLatency (the
setpoint actually resolved — nothing exposed this before), the cost counters
catchUpSkips / framesDropped / audioUnderruns published by a new always-on
trackPlayoutHealth, TrackSubscriberContext.arrivalJitter (a decaying
arrival-offset envelope, whose spread is the jitter primitive), and MSF jitter
(§5.2.9) surviving catalog parsing.
Caveat, stated plainly: the adaptive constants are reasoned, not measured
This matters for review, so it should not be buried. The defect fixes above are
backed by observed behaviour. The adaptive controller's tuning constants are not:
maxWidenRatePerSecond = 0.02s/smaxNarrowRatePerSecond = 0.005s/s (a quarter of the widen rate)widenStep = 0.15s,widenMax = 1s,widenDecayPerSecond = 0.01- Both rates strictly below
latency.clockSlewRate / 2(0.025 by default), behind a
Schmitt trigger sized off the finest band any inner loop acts on.
These were derived from a separation-of-timescales argument, not from measurement.
The argument is that three negative-feedback loops act on one variable — the
bang-bang rateNudge, the self-clock's continuous slew, and this behavior's moving
setpoint — and a setpoint that moves faster than the inner loops can follow is a
setpoint they never reach. The bound is validated rather than merely documented:
resolveAdaptiveLatencyConfig derives its defaults from the latency config so an
untouched adaptive config can never violate it, and throws on an explicit value
that does.
But the specific numbers inside that bound are a judgement call. floorSeconds is
likewise a flat constant, not a derivation (hang's precedent is the same shape — a
flat 100ms fallback). A reviewer should treat them as a starting point to be tuned
against real paths, and that is a large part of why the whole thing ships off by
default.
There is also a hard observability limit worth knowing: there is no transport RTT
available. spf speaks its own draft-19 stack (no PROBE — that is a moq-lite
extension) and browsers do not populate WebTransport.getStats().smoothedRtt, so
absolute one-way delay simply cannot be measured. The estimate is built from arrival
jitter spread, the publisher's declared jitter, a constant cushion, and failure
feedback.
Relationship to the autoplay work
Integrated, not redundant — this is likely the first question, so: the three
autoplay commits (23d3ef62, cd36ce9a, 0817a879) touch none of the five
files this work centres on.
| File this work centres on | Touched by autoplay commits? |
|---|---|
packages/spf/src/playback/behaviors/sync-latency.ts | no |
packages/spf/src/playback/actors/dom/audio-renderer.ts | no |
packages/spf/src/playback/actors/dom/video-renderer.ts | no |
packages/spf/src/playback/behaviors/dom/setup-moq-renderers.ts | no |
packages/spf/src/media/moq/timeline.ts | no |
Where the two do meet, the change is purely additive:
| Shared file | This PR |
|---|---|
packages/spf/src/playback/engines/moq/engine.ts | +59 / −6 |
packages/spf/src/playback/engines/moq/adapter.ts | +33 / −0 |
packages/html/src/media/simple-moq-video/index.ts | +8 / −1 |
packages/html/src/media/simple-moq-video/tests/index.test.ts | +30 / −0 |
subscribe-selected-tracks.ts, its tests, and moq/tests/adapter.test.ts — the rest
of the autoplay surface — are untouched entirely.
Concretely:
adaptive-latencysits besideautoplayinobservedAttributesand copies its
presence-based semantics deliberately, so the two attributes behave the same way.- The renderers handle the deferred-audio gate as a named case, not as an
accident. A deferred-audio unlock creates a fresh subscription whose edge can land
behind the video self-clock, which only re-anchors on forward discontinuities and
would otherwise freeze until the new master clock caught up — so the audio join
anchor is clamped forward to a running video clock. suspendMediaWhilePausedand theaudioSuspendedgate both rejoin at the live
edge, which is the join-at-edge path this PR fixes.
Tests
1449 passing, up +39 from the 1410 baseline on mmcc/moq-sandbox.
New coverage was added as siblings of the existing fixed-setpoint tests rather
than as edits to them, so every assertion pinning the untouched chain still runs
verbatim. No autoplay test file loses a line (see the table above). The 28 removed
lines across three spf test files are helper extraction and import rewrites — for
example, the currentTime fallback test moved describe blocks because publishing
currentTime moved from setupAudioRenderer to trackPlayoutTime. No assertion was
deleted or weakened.
New test files:
packages/spf/src/playback/behaviors/tests/adapt-latency-target.test.ts(+410)packages/spf/src/playback/actors/dom/tests/video-renderer.test.ts(+305)packages/spf/src/playback/actors/dom/tests/audio-renderer.test.ts(+134)
Verified locally on this branch
| Command | Result |
|---|---|
pnpm install --frozen-lockfile | exit 0 — lockfile unchanged, "Already up to date" |
pnpm lint | exit 0 (one pre-existing warning in resolve-catalog.ts, a file this PR does not touch) |
pnpm build:packages | exit 0 — 10/10 tasks |
pnpm typecheck | exit 0 |
pnpm turbo run test --filter="@videojs/spf" --env-mode=loose | exit 0 — 1449 passed, 14 skipped, 95 files passed / 1 skipped, no type errors |
pnpm check:workspace | exit 1 — 9 passed, 1 failed: .claude/plans only (see below) |
Known CI failure, pre-existing
The workspace job (pnpm check:workspace) will fail with:
.claude/plans: must resolve to .agents/plans
This is not caused by this PR. .claude/plans/moq-engine-implementation.md is a
tracked file added in 03cdae43 and modified by the autoplay commit 23d3ef62;
check:workspace expects .claude/plans to be a generated alias of .agents/plans,
and a real tracked directory there breaks that invariant. pnpm install says the
same thing at postinstall time: warning: refusing to replace non-generated .claude/plans.
Verified pre-existing: running pnpm check:workspace on origin/mmcc/moq-sandbox
(0817a879) with none of this PR's commits applied produces the identical result —
9 passed, 1 failed, same message.
Left alone deliberately — it is the sandbox branch owner's call whether the file
moves to .agents/plans or the check gets an exemption.
Summary by cubic
Fixes MoQ playout timing to hold steady near the live edge and adds an optional adaptive-latency controller (off by default). Also hardens session/protocol handling and validates audio config to make playback safer and more predictable.
-
Bug Fixes
- Join at the live edge: anchor at newest buffered − target instead of the oldest frame; re-anchor after catch-up skips.
- Measure real latency: compare delivery edge to
state.currentTime(not jitter-buffer depth). Video self-clock now slews toward the edge continuously with a bounded rate; prevents drift and join-burst lock-in. - Move playout time publishing to
trackPlayoutTimeso the controller stays active in video-only cases; fix slew budget carryover when audio ends. - Protocol hardening: truncated control messages normalize to
MoqtProtocolError; request timeouts cancel request streams so lateSUBSCRIBE_OKcannot resurrect routes; pendingTRACK_STATUSrequests tracked and failed on teardown; raiseMAX_OBJECT_PROPERTIES_LENGTHto 1 MiB. - Catalog/codec validation:
parseChannelConfigrequires a safe, finite total and handles dotted layouts;toAudioDecoderConfigderivesnumberOfChannelsfromchannelConfig, validates positive integersampleRate, and rejects bad tracks. - Pause/resume robustness:
suspendMediaWhilePausedderives its hold window from the effective target (explicit or adaptive) and ignores non-finite/negative inputs; resume paths chain behind an in-flight suspend;resolveCatalogignores joining-fetch entries after the settle deadline and guards callbacks by attempt. simple-moq-video: report 0×0 dimensions before the first frame (canvases default to 300×150); drop NaN volume writes and clamp infinities.
-
New Features
- Optional adaptive target latency via
adaptLatencyTargetwith precedence: explicitstate.targetLatency>state.adaptiveTargetLatency> catalog → default. Disabled unlessMoqEngineConfig.adaptiveLatency.enabledorsimple-moq-videosetsadaptive-latency. - New observability:
state.effectiveTargetLatency,catchUpSkips,framesDropped,audioUnderruns,TrackSubscriberContext.arrivalJitter, and catalogjitterparsed and exposed. - Renderer setup passes edge anchors from subscriber stats; audio discards pre-anchor backlog, video decodes backlog and drops late; deferred-audio unlock clamps forward to a running video clock.
- New helpers:
resolveTargetLatencySecondsandpreferredTargetLatencySecondsunify target resolution across consumer input, adaptive proposal, and catalog. - HTML element:
simple-moq-videoacceptsadaptive-latencyattribute, routed to the engine without altering numerictarget-latency.
- Optional adaptive target latency via