feat(plugin): implement YouTube WASM plugin via yt-dlp
Merged
MP
7mo ago+2940+2940+0123456789012345678901234567890123456789-106-106-012345678901234567890123456789
Summary
- Adds
plugins/vortex-mod-youtube/, the first production crawler plugin for Vortex, targetingwasm32-wasip1viaextism-pdk1.4. - Resolves YouTube videos, Shorts, playlists, and channels by delegating media extraction to
yt-dlpthrough the hostrun_subprocessfunction — no direct network I/O from the plugin sandbox. - Closes task 24 from
specs/01-mvp/README.md.
Architecture
Native-testable core / WASM-thin-wrapper split. The plugin keeps business logic in plain Rust modules that compile for the host toolchain, and gates the #[plugin_fn] exports / #[host_fn] imports behind cfg(target_family = "wasm"). Result: 77 unit tests run under cargo test --target x86_64-unknown-linux-gnu without spinning up a WASM runtime, while the release build still produces a 1.2 MB stripped .wasm binary.
| Module | Purpose |
|---|---|
url_matcher | Regex-based classification of youtube.com / youtu.be / /shorts/ / /playlist / /@handle / /channel/ / /user/ / /c/ URLs; video + playlist ID extraction. |
metadata | Serde parsing of yt-dlp --dump-json (single video) and --flat-playlist (accepts both JSONL and envelope shapes). Classifies formats as muxed / video_only / audio_only / unknown via vcodec/acodec inspection. |
quality_manager | Format selection by target resolution (360p → 4320p + Best) with height-bucket fallback and user container preference. Audio-only path picks highest abr. |
extractor | Pure helpers that build subprocess request JSON and parse the response envelope. UTF-8-safe stderr truncation. |
plugin_api (WASM-only) | #[plugin_fn] exports: can_handle, supports_playlist, extract_links, get_media_variants, extract_playlist. #[host_fn] extern \"ExtismHost\" import of run_subprocess. |
error | thiserror enum with SerdeJson(#[from]) preserving the error source chain. |
Security hardening (applied from code review)
--sentinel inserted before every URL argument passed toyt-dlp, so a URL that accidentally starts with-can never be interpreted as an option. Defense in depth beyond host-side URL validation.- Stderr truncation operates on character boundaries rather than byte offsets — multi-byte yt-dlp output (non-ASCII filenames, localised errors) can no longer cause a WASM trap.
- Least-privilege manifest: only
subprocess = [\"yt-dlp\"]is declared; thehttpcapability is omitted because all network access is delegated to yt-dlp. - The sole
unsafeblock (therun_subprocesshost-function call) documents four named invariants: host symbol resolution, ABI shape, capability check, and ownership. - Zero
.unwrap()/.expect()in production paths (CLAUDE.md rule).build_subprocess_requestreturnsResulteven though the failure branch is practically unreachable. - Flat-playlist parser accepts both JSONL (typical for playlists) and envelope (sometimes emitted for channels) shapes — prior fail-fast behaviour would have broken every channel URL.
Test plan
-
cargo test --target x86_64-unknown-linux-gnu— 77/77 pass (url_matcher: 22, metadata: 16, quality_manager: 13, extractor: 10, ipc_tests: 10, error: 0 new) -
cargo clippy --target x86_64-unknown-linux-gnu --all-targets -- -D warnings— clean -
cargo clippy --all-targets -- -D warnings(WASM target) — clean -
cargo fmt -- --check— clean -
cargo build --release— producestarget/wasm32-wasip1/release/vortex_mod_youtube.wasm(1.2 MB) - lefthook
pre-push(src-tauriworkspace cargo test + vitest + tsc) — clean - Manual smoke test: copy
plugin.tomland the release.wasminto~/.config/vortex/plugins/vortex-mod-youtube/, ensureyt-dlpis onPATH, then verifyplugin_loader.resolve_url("https://youtu.be/...")routes to the plugin in a running Vortex build. - Manual smoke test: trigger
extract_linksandget_media_variantsend-to-end once a driving adapter wires them in.
Follow-ups (not in this PR)
- Wire
extract_links/get_media_variants/extract_playlistcalls from the Vortex application layer — currently onlycan_handleis consumed byresolve_url. - Graceful partial-parse for malformed lines in very long playlists (currently fails fast on the first bad entry).
- Collapse
watch_path_regexandwatch_id_regexinto a single compiled pattern.
Summary by cubic
Adds a hardened YouTube WASM plugin via yt-dlp that handles videos, Shorts, playlists, and channels with strict URL validation and robust parsing. Completes MVP task 24.
-
New Features
- New
plugins/vortex-mod-youtubecrate forwasm32-wasip1usingextism-pdk1.4; exportscan_handle,supports_playlist,extract_links,get_media_variants,extract_playlist. - Delegates extraction to
yt-dlp(--dump-json); supports quality/container choice and audio-only; variants listed from metadata. - Hardened URL handling: explicit YouTube host allowlist (incl.
youtube-nocookie.com,youtu.be); path-based regex matching; trims userinfo/ports; rejects query/fragment smuggling; preserves case of IDs; improved ID extraction for ports/userinfo. - Robust parsing: playlist parser accepts JSONL and envelope; canonical watch-URL fallback when missing; guards invalid durations;
get_media_variantsrejects playlists;extract_playlistrejects single videos;can_handlerejects unsupported paths like/embed/. - Security/quality:
--sentinel before URLs; least-privilege manifest (subprocess = ["yt-dlp"], no HTTP); UTF-8-safe stderr truncation; 105 native tests; release.wasm~1.2 MB.
- New
-
Migration
- Ensure
yt-dlpis onPATHand host allowssubprocessforyt-dlp. - Build for
wasm32-wasip1, then install to~/.config/vortex/plugins/vortex-mod-youtube/withplugin.tomland exactly one.wasmfile (rename tovortex-mod-youtube.wasmif needed). - Optional
plugin.tomlconfig:default_quality(incl.best),preferred_format,audio_only, subtitles,cookie_file.
- Ensure
Summary by CodeRabbit
-
New Features
- Added a Vortex YouTube WASM plugin: supports videos, playlists, channels, and Shorts; media variant listing with selectable quality (360p–4320p/Best), container preference, and audio-only mode. Plugin runs yt-dlp via host subprocess.
-
Documentation
- Added plugin README and manifest with install, build, runtime, config and host prerequisites.
-
Tests
- Added extensive native unit tests for URL classification, parsing, IPC helpers, and quality/format selection.
-
Chores
- Optimized WASM release build/profile for size.