feat(plugin): implement YouTube WASM plugin via yt-dlp

Merged
MP
7mo ago
+2940-106

Summary

  • Adds plugins/vortex-mod-youtube/, the first production crawler plugin for Vortex, targeting wasm32-wasip1 via extism-pdk 1.4.
  • Resolves YouTube videos, Shorts, playlists, and channels by delegating media extraction to yt-dlp through the host run_subprocess function — no direct network I/O from the plugin sandbox.
  • Closes task 24 from specs/01-mvp/README.md.

Architecture

Native-testable core / WASM-thin-wrapper split. The plugin keeps business logic in plain Rust modules that compile for the host toolchain, and gates the #[plugin_fn] exports / #[host_fn] imports behind cfg(target_family = "wasm"). Result: 77 unit tests run under cargo test --target x86_64-unknown-linux-gnu without spinning up a WASM runtime, while the release build still produces a 1.2 MB stripped .wasm binary.

ModulePurpose
url_matcherRegex-based classification of youtube.com / youtu.be / /shorts/ / /playlist / /@handle / /channel/ / /user/ / /c/ URLs; video + playlist ID extraction.
metadataSerde parsing of yt-dlp --dump-json (single video) and --flat-playlist (accepts both JSONL and envelope shapes). Classifies formats as muxed / video_only / audio_only / unknown via vcodec/acodec inspection.
quality_managerFormat selection by target resolution (360p → 4320p + Best) with height-bucket fallback and user container preference. Audio-only path picks highest abr.
extractorPure helpers that build subprocess request JSON and parse the response envelope. UTF-8-safe stderr truncation.
plugin_api (WASM-only)#[plugin_fn] exports: can_handle, supports_playlist, extract_links, get_media_variants, extract_playlist. #[host_fn] extern \"ExtismHost\" import of run_subprocess.
errorthiserror enum with SerdeJson(#[from]) preserving the error source chain.

Security hardening (applied from code review)

  • -- sentinel inserted before every URL argument passed to yt-dlp, so a URL that accidentally starts with - can never be interpreted as an option. Defense in depth beyond host-side URL validation.
  • Stderr truncation operates on character boundaries rather than byte offsets — multi-byte yt-dlp output (non-ASCII filenames, localised errors) can no longer cause a WASM trap.
  • Least-privilege manifest: only subprocess = [\"yt-dlp\"] is declared; the http capability is omitted because all network access is delegated to yt-dlp.
  • The sole unsafe block (the run_subprocess host-function call) documents four named invariants: host symbol resolution, ABI shape, capability check, and ownership.
  • Zero .unwrap() / .expect() in production paths (CLAUDE.md rule). build_subprocess_request returns Result even though the failure branch is practically unreachable.
  • Flat-playlist parser accepts both JSONL (typical for playlists) and envelope (sometimes emitted for channels) shapes — prior fail-fast behaviour would have broken every channel URL.

Test plan

  • cargo test --target x86_64-unknown-linux-gnu — 77/77 pass (url_matcher: 22, metadata: 16, quality_manager: 13, extractor: 10, ipc_tests: 10, error: 0 new)
  • cargo clippy --target x86_64-unknown-linux-gnu --all-targets -- -D warnings — clean
  • cargo clippy --all-targets -- -D warnings (WASM target) — clean
  • cargo fmt -- --check — clean
  • cargo build --release — produces target/wasm32-wasip1/release/vortex_mod_youtube.wasm (1.2 MB)
  • lefthook pre-push (src-tauri workspace cargo test + vitest + tsc) — clean
  • Manual smoke test: copy plugin.toml and the release .wasm into ~/.config/vortex/plugins/vortex-mod-youtube/, ensure yt-dlp is on PATH, then verify plugin_loader.resolve_url("https://youtu.be/...") routes to the plugin in a running Vortex build.
  • Manual smoke test: trigger extract_links and get_media_variants end-to-end once a driving adapter wires them in.

Follow-ups (not in this PR)

  • Wire extract_links / get_media_variants / extract_playlist calls from the Vortex application layer — currently only can_handle is consumed by resolve_url.
  • Graceful partial-parse for malformed lines in very long playlists (currently fails fast on the first bad entry).
  • Collapse watch_path_regex and watch_id_regex into a single compiled pattern.

Summary by cubic

Adds a hardened YouTube WASM plugin via yt-dlp that handles videos, Shorts, playlists, and channels with strict URL validation and robust parsing. Completes MVP task 24.

  • New Features

    • New plugins/vortex-mod-youtube crate for wasm32-wasip1 using extism-pdk 1.4; exports can_handle, supports_playlist, extract_links, get_media_variants, extract_playlist.
    • Delegates extraction to yt-dlp (--dump-json); supports quality/container choice and audio-only; variants listed from metadata.
    • Hardened URL handling: explicit YouTube host allowlist (incl. youtube-nocookie.com, youtu.be); path-based regex matching; trims userinfo/ports; rejects query/fragment smuggling; preserves case of IDs; improved ID extraction for ports/userinfo.
    • Robust parsing: playlist parser accepts JSONL and envelope; canonical watch-URL fallback when missing; guards invalid durations; get_media_variants rejects playlists; extract_playlist rejects single videos; can_handle rejects unsupported paths like /embed/.
    • Security/quality: -- sentinel before URLs; least-privilege manifest (subprocess = ["yt-dlp"], no HTTP); UTF-8-safe stderr truncation; 105 native tests; release .wasm ~1.2 MB.
  • Migration

    • Ensure yt-dlp is on PATH and host allows subprocess for yt-dlp.
    • Build for wasm32-wasip1, then install to ~/.config/vortex/plugins/vortex-mod-youtube/ with plugin.toml and exactly one .wasm file (rename to vortex-mod-youtube.wasm if needed).
    • Optional plugin.toml config: default_quality (incl. best), preferred_format, audio_only, subtitles, cookie_file.

Summary by CodeRabbit

  • New Features

    • Added a Vortex YouTube WASM plugin: supports videos, playlists, channels, and Shorts; media variant listing with selectable quality (360p–4320p/Best), container preference, and audio-only mode. Plugin runs yt-dlp via host subprocess.
  • Documentation

    • Added plugin README and manifest with install, build, runtime, config and host prerequisites.
  • Tests

    • Added extensive native unit tests for URL classification, parsing, IPC helpers, and quality/format selection.
  • Chores

    • Optimized WASM release build/profile for size.