feat(spf): opt-in MoQ audio renderer over the shared worklet audio core

Merged
MM
1mo ago
+1235-24

Refs #66. This is the unblocked part of the plan in https://github.com/mmcc/v10/issues/66#issuecomment-6273710149. It does not close #66.

What

createWorkletAudioRendererActor puts the MoQ audio renderer on top of the shared AudioWorklet audio core, @mux/moq-media's createAudioPlayer from muxinc/relay-sdks#323. It sits behind the existing AudioRendererActor seam, so the video renderer, trackPlayoutTime, syncLatency and adaptLatencyTarget are unchanged.

The engine takes the core as createAudioPlayer in its config. When it is absent, the AudioBufferSourceNode renderer stays the default.

  • Seam. The renderer types against a structural interface that mirrors the parts of AudioPlayer, AudioPlayerInput and PlayoutClock it uses. @mux/moq-media is not a dependency (see below).
  • Metering. Frames move from the subscriber's jitter buffer into input.push so the worklet holds about its own jitterTargetUs. v10's target latency stays in the subscriber buffer. The core's adaptive target is left alone, and the target latency v10 resolves caps its ceiling through clock.targetLatencyUs.
  • Join-at-edge anchor. Frames behind getJoinAnchorUs() are dropped before the first push. A timeline jump over 1 s calls input.cut() and re-arms the anchor. When a re-join's whole buffer is already past the anchor, the clock is placed at the anchor and the head is held back until it is due.
  • Clock. The renderer owns the clock the core reports into, so getClockTimeUs() keeps its contract: undefined before audio, held where the worklet ran dry (not wall-paced), and resuming seamlessly when audio does. Reports of audio that was not pushed on the current timeline are ignored.
  • Underruns. Counted on the renderer's own dry edges, as audio-renderer counts them, across track changes. The core's stats.underruns skips stalls longer than its jitter ceiling, which adaptLatencyTarget still needs as failure evidence.
  • Adapter. The render context now forwards baseLatency and outputLatency, and exposes baseContext, the real context an AudioWorkletNode needs.

Why opt-in

  1. No rate control in the core. v10 builds and sheds depth with ±5 % playoutRate nudges. The core can only trim or run dry, so this path ignores nudges. I filed muxinc/relay-sdks#329 for AudioPlayerInput.playbackRate.
  2. No public package yet. muxinc/relay-sdks is private and this repo is public, so vendoring the core would publish unreleased code. The npm release is tracked in muxinc/relay-sdks#222 and #224.

Verification

  • pnpm -F @videojs/spf test: 2410 passed, 11 skipped. That includes 9 new createWorkletAudioRendererActor tests against a fake core and a setupAudioRenderer selection test.
  • tsgo -p packages/spf/tsconfig.json: clean.
  • vp lint / vp fmt on the changed files: no new warnings.
  • Headless Chromium, one browser at a time. I used the sandbox's loopback relay with the real core copied from relay-sdks 140b8a0 into the gitignored apps/sandbox/src/; none of it is committed. Results over 12 s:
Worklet coreSource nodes
Clock owner, rateaudio, 1.00×audio, 1.00×
Underruns00
OutputRMS ≈ 0.01, ~50 reports/s, jitter target 60 → 50 ms—
Measured latency0.145 s for the whole run, nudging at 0.950.12 → 0.48 s, stable after 7.5 s

The latency row is gap 1 measured live. A fresh join plays at the edge, and only a rate nudge can build depth back to the 500 ms target. Until #329 lands, the worklet path stays nudging, which also freezes adaptLatencyTarget.

Follow-up (#66)

When @mux/moq-media is published with playbackRate: add the dependency, apply getPlaybackRate() to input.playbackRate, make the core the default, and delete the AudioBufferSourceNode path.

🤖 Generated with Claude Code


Summary by cubic

Adds an opt-in MoQ audio renderer over the shared AudioWorklet audio core, @mux/moq-media's createAudioPlayer, behind the existing AudioRendererActor seam. When no createAudioPlayer is configured in the engine config, the AudioBufferSourceNode renderer stays the default.

Frames move from the subscriber's jitter buffer into the worklet so the core holds about its own jitter target, while v10's target latency stays in the subscriber buffer and caps the core's ceiling via the clock. The video renderer, trackPlayoutTime, syncLatency, and adaptLatencyTarget are unchanged. The clock keeps its contract — undefined before audio, held where the worklet ran dry — and never steps back in recovery: the hold releases only when the core reports itself stopped or a report passes the dry point, and a floor keeps the clock at the held position until the refill's first report reaches it. A placed re-join clock stops at the end of the audio pushed and holds there until the core plays, and a frozen placement refills like an underrun so a gap past its fill horizon cannot strand it. Reports are timed on AudioContext.currentTime, so the clock freezes through pause and resumes without jumping, and reports from removed inputs are ignored. Underruns count every time the worklet runs dry, as audio-renderer counts them, including a dry edge that lands on a track change before the next tick. Frame durations are read from each Opus packet's TOC byte, falling back to the shortest step between frames, so gaps of lost packets don't extend pushed audio past where it ends.

Why opt-in

  • The core has no playback rate yet (muxinc/relay-sdks#329), so playoutRate nudges are ignored on this path.
  • @mux/moq-media is unreleased and this repo is public, so the core is injected as a structural interface rather than a dependency.