Add POST /api/songs/moments: sections and moments measured from the audio

Merged
SI
2d ago
+1963-3

What this adds

A new endpoint, POST /api/songs/moments, that finds a song's sections (intro, verse, pre-chorus, chorus, bridge, outro) and its best short moments for licensing, each with exact times.

Times come from measuring the audio, not from a language model. Music Flamingo guessed song lengths 20–30 s wrong and put choruses on a 30-second grid in testing, so it no longer locates anything. A new CPU-only Modal app (song-moments) measures the beat grid, loudness and which passages repeat. Music Flamingo then describes each moment from a clip of just that moment (mood, best uses, vocals).

  • Section names: pass section_order (the headers from a lyric sheet, in order) and the sections are named from it. If the sheet does not fit the audio (its choruses don't sound alike), names come from repetition and loudness instead, and names_from says which.
  • Moments: vocal_hook, chorus_lift, build_up, climax, breakdown, intro, outro. Each carries :06, :15 and :30 cuts that start on the beat. Overlapping moments merge (also).
  • Song-level: tempo, integrated loudness (LUFS), fade vs. clean ending, and a per-second energy curve.
  • describe: true (default) describes every moment; false or a list of moment types limits the model calls.

Pricing

$0.01 per song measured (a few CPU seconds on Modal), plus the existing /api/songs/analyze rate for each moment described. The pre-flight credit gate covers the measurement plus one model base fee per moment that may be described.

Changes

  • modal/song_moments.py: the measurement (pure Python, runs locally too). modal/serve_song_moments.py: the Modal app. Deploy step added to deploy-modal.yml.
  • modal/serve_music_flamingo.py: optional clip_start/clip_end on generate, which cuts the clip with ffmpeg before inference. Whole-song calls are unchanged.
  • lib/songMoments/*: validation, the Modal client, moment descriptions, charging and the handler, mirroring /api/songs/analyze. callFlamingoGenerate takes an internal clip option; the public analyze schema is unchanged.

Tests

  • pnpm vitest run lib/songMoments lib/flamingo: 27 files, 179 tests pass.
  • modal/test_song_moments.py (synthetic song with known section times): 5 pass. Run it with uv (see modal/README.md).
  • 50 Seeker catalog songs were measured locally with the same code. All 50 completed in about 1.5 s each. 33 of 42 lyric sheets fit, and 46 songs got a hook. A listen-check by ear is in progress.
  • Not yet verified: the deployed Modal endpoint and the clip descriptions. Both deploy on merge; the 50 songs will be rerun through the live endpoint after that.

🤖 Generated with Claude Code

Summary by cubic

Adds POST /api/songs/moments, which finds a song's sections (intro, verse, pre-chorus, chorus, bridge, outro) and its best short moments for licensing, each with exact times measured from the audio on a new CPU-only Modal app (song-moments). Music Flamingo no longer locates anything—it only describes each moment from a clip of just that moment, cut on the model's side via optional clip_start/clip_end.

Changes

  • Sections are named from the lyric sheet's section_order when it fits the audio; otherwise from repetition and loudness, with names_from indicating the source.
  • Moments (vocal_hook, chorus_lift, build_up, climax, breakdown, intro, outro) carry :06, :15, and :30 cuts that start on the beat; overlapping moments merge via also.
  • Song-level results include tempo, loudness (LUFS), ending type, a per-second energy curve, and a measured_with version so stored results can be rerun when the measurement changes.
  • describe defaults to true; false or a list of moment types limits model calls.
  • Pricing: $0.01 per song measured plus the existing /api/songs/analyze rate per described moment; the credit gate covers both.
  • New Modal deploy step in deploy-modal.yml; song_moments.py is pure Python and runs locally.