feat(spf): render MoQ audio through the @caprid/moq-media worklet core

Merged
MM
25d ago
+1050-2097

Closes #66

Summary

The relay SDKs are now on npm as @caprid/*, so this makes the AudioWorklet player core from @caprid/moq-media the only MoQ audio renderer and deletes the AudioBufferSourceNode path. The latency controller's rate nudges now reach the core, which was the last reason the core was opt-in.

Changes

  • @caprid/moq-media ^0.1.0 is now a dependency of @videojs/spf. @caprid/* is added to minimumReleaseAgeExclude, because 0.1.0 was published the day before.
  • The worklet renderer replaces audio-renderer.ts and keeps the createAudioRendererActor name. The source-node renderer and its tests are removed.
  • playoutRate reaches the core as the input's playbackRate, clamped to the 0.5–2 range the core accepts. A new input starts at the current rate. On each change the clock first saves its position at the old speed, so it always moves at the rate in effect. The output-latency term is converted to media time at the current rate.
  • With no factory given, the renderer builds the core on the render context's baseContext. Without one, the renderer goes to an error status. The engine's createAudioPlayer option stays, but only as an override.
  • AudioContextLike drops createBuffer/createBufferSource, and so do the adapter's render context and the test fakes.
  • The renderer's local Opus duration helper is replaced by the package's opusPacketDurationUs. Comments that referred to @mux/moq-media or "the relay SDKs" now name the @caprid packages.

Testing

  • pnpm -F @videojs/spf test: 2414 passed. pnpm typecheck, format and pnpm check:workspace pass. The simple-moq-video html tests pass.
  • New unit tests:
    • the rate reaches the input, and the clock speeds up only after the change (100 ms, then 1.05× → 205 ms);
    • rates outside 0.5–2 are clamped, and a new input starts at the current rate;
    • the default factory loads the worklet onto baseContext, and goes to an error status without one.
  • A one-off check in headless Chromium (not committed) used the real core, a real AudioContext, and 6 s of Opus encoded with WebCodecs:
    • first audio after 41 ms;
    • clock speed 1.003× the context clock at rate 1, and 1.056× after a 1.05 nudge;
    • 0 underruns, and frames were metered from the source buffer rather than pushed all at once.
  • Bundle (moq entry, esbuild, minified): 102.9 → 111.6 KB, 33.1 → 36.5 KB gzipped. pnpm -F @videojs/spf size is broken on mmcc/moq (it calls pnpm build, which the Vite+ migration removed), so I measured with esbuild instead.

Follow-up, not in this PR: the dry and hold detection could read the core's dryAtUs and playStarts stats instead of working them out from timestamps.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DaH4ihf1LLPqmXShYkvBat


Summary by cubic

Makes @caprid/moq-media's AudioWorklet player core the only MoQ audio renderer, deleting the AudioBufferSourceNode path, and sends the latency controller's playout rate into the core, which was the last reason the worklet renderer was opt-in.

  • Adds @caprid/moq-media ^0.1.0 as a dependency and @caprid/* to minimumReleaseAgeExclude.
  • The worklet renderer replaces audio-renderer.ts; the source-node renderer and its tests are removed.
  • playoutRate reaches the core as playbackRate, clamped to the 0.5–2 range, and the clock rebases on each change so it advances at the rate in effect.
  • With no override given, the renderer builds the core on the render context's baseContext; without one it goes to an error status. The engine's createAudioPlayer option stays as an override.
  • Packets resuming past a re-join gap now start a fresh fill instead of arriving one per stall rescue, which froze the clock for up to the core's jitter ceiling.
  • AudioContextLike drops createBuffer/createBufferSource, and the local Opus duration helper is replaced by the package's opusPacketDurationUs.