feat(spf): opt-in MoQ audio renderer over the shared worklet audio core
Refs #66. This is the unblocked part of the plan in https://github.com/mmcc/v10/issues/66#issuecomment-6273710149. It does not close #66.
What
createWorkletAudioRendererActor puts the MoQ audio renderer on top of the shared AudioWorklet audio core, @mux/moq-media's createAudioPlayer from muxinc/relay-sdks#323. It sits behind the existing AudioRendererActor seam, so the video renderer, trackPlayoutTime, syncLatency and adaptLatencyTarget are unchanged.
The engine takes the core as createAudioPlayer in its config. When it is absent, the AudioBufferSourceNode renderer stays the default.
- Seam. The renderer types against a structural interface that mirrors the parts of
AudioPlayer,AudioPlayerInputandPlayoutClockit uses.@mux/moq-mediais not a dependency (see below). - Metering. Frames move from the subscriber's jitter buffer into
input.pushso the worklet holds about its ownjitterTargetUs. v10's target latency stays in the subscriber buffer. The core's adaptive target is left alone, and the target latency v10 resolves caps its ceiling throughclock.targetLatencyUs. - Join-at-edge anchor. Frames behind
getJoinAnchorUs()are dropped before the first push. A timeline jump over 1 s callsinput.cut()and re-arms the anchor. When a re-join's whole buffer is already past the anchor, the clock is placed at the anchor and the head is held back until it is due. - Clock. The renderer owns the clock the core reports into, so
getClockTimeUs()keeps its contract:undefinedbefore audio, held where the worklet ran dry (not wall-paced), and resuming seamlessly when audio does. Reports of audio that was not pushed on the current timeline are ignored. - Underruns. Counted on the renderer's own dry edges, as
audio-renderercounts them, across track changes. The core'sstats.underrunsskips stalls longer than its jitter ceiling, whichadaptLatencyTargetstill needs as failure evidence. - Adapter. The render context now forwards
baseLatencyandoutputLatency, and exposesbaseContext, the real context anAudioWorkletNodeneeds.
Why opt-in
- No rate control in the core. v10 builds and sheds depth with ±5 %
playoutRatenudges. The core can only trim or run dry, so this path ignores nudges. I filed muxinc/relay-sdks#329 forAudioPlayerInput.playbackRate. - No public package yet.
muxinc/relay-sdksis private and this repo is public, so vendoring the core would publish unreleased code. The npm release is tracked in muxinc/relay-sdks#222 and #224.
Verification
pnpm -F @videojs/spf test: 2410 passed, 11 skipped. That includes 9 newcreateWorkletAudioRendererActortests against a fake core and asetupAudioRendererselection test.tsgo -p packages/spf/tsconfig.json: clean.vp lint/vp fmton the changed files: no new warnings.- Headless Chromium, one browser at a time. I used the sandbox's loopback relay with the real core copied from relay-sdks
140b8a0into the gitignoredapps/sandbox/src/; none of it is committed. Results over 12 s:
| Worklet core | Source nodes | |
|---|---|---|
| Clock owner, rate | audio, 1.00× | audio, 1.00× |
| Underruns | 0 | 0 |
| Output | RMS ≈ 0.01, ~50 reports/s, jitter target 60 → 50 ms | — |
| Measured latency | 0.145 s for the whole run, nudging at 0.95 | 0.12 → 0.48 s, stable after 7.5 s |
The latency row is gap 1 measured live. A fresh join plays at the edge, and only a rate nudge can build depth back to the 500 ms target. Until #329 lands, the worklet path stays nudging, which also freezes adaptLatencyTarget.
Follow-up (#66)
When @mux/moq-media is published with playbackRate: add the dependency, apply getPlaybackRate() to input.playbackRate, make the core the default, and delete the AudioBufferSourceNode path.
🤖 Generated with Claude Code
Summary by cubic
Adds an opt-in MoQ audio renderer over the shared AudioWorklet audio core, @mux/moq-media's createAudioPlayer, behind the existing AudioRendererActor seam. When no createAudioPlayer is configured in the engine config, the AudioBufferSourceNode renderer stays the default.
Frames move from the subscriber's jitter buffer into the worklet so the core holds about its own jitter target, while v10's target latency stays in the subscriber buffer and caps the core's ceiling via the clock. The video renderer, trackPlayoutTime, syncLatency, and adaptLatencyTarget are unchanged. The clock keeps its contract — undefined before audio, held where the worklet ran dry — and never steps back in recovery: the hold releases only when the core reports itself stopped or a report passes the dry point, and a floor keeps the clock at the held position until the refill's first report reaches it. A placed re-join clock stops at the end of the audio pushed and holds there until the core plays, and a frozen placement refills like an underrun so a gap past its fill horizon cannot strand it. Reports are timed on AudioContext.currentTime, so the clock freezes through pause and resumes without jumping, and reports from removed inputs are ignored. Underruns count every time the worklet runs dry, as audio-renderer counts them, including a dry edge that lands on a track change before the next tick. Frame durations are read from each Opus packet's TOC byte, falling back to the shortest step between frames, so gaps of lost packets don't extend pushed audio past where it ends.
Why opt-in
- The core has no playback rate yet (muxinc/relay-sdks#329), so
playoutRatenudges are ignored on this path. @mux/moq-mediais unreleased and this repo is public, so the core is injected as a structural interface rather than a dependency.