Song moments: extended measurement (loudness, bands, fingerprint, drops, edit points)

Merged
SI
2d ago
+1188-24

Stacked on #1225 (codex/song-moments); review and merge that first.

What

Extends the CPU song-moments measurement with every number a song report draws. No language model.

  • modal/song_measure.py (new): measure_extended(y_stereo, sr, base, tempo_hint=None, audio_path=None) returns the base measurement plus lufs, true_peak_dbtp, sample_peak_dbfs, crest_factor_db, loudness_range_lu, broadcast_ready, stereo, spectral_centroid_hz, rolloff95_hz, band_profile, fingerprint, fingerprint_inputs, key, time_signature, valence, arousal, mute_drops, chord_loop, edit_points, tempo_note, and measured_with. Small pure helpers per concern; the fingerprint keeps its calibration comments.
  • song_moments.measure_file gains tempo_hint and returns the extended measurement (measure_song is unchanged).
  • serve_song_moments.py accepts optional tempo_hint (30–300 BPM, 422 otherwise), decodes stereo at 44.1 kHz, runs ffmpeg ebur128 on the downloaded file before deleting it, and returns the extended fields. sections, moments, energy_curve and ending stay at the top level. The pip list needed no change (pyloudnorm was already there; ffmpeg was already in the image). MEMORY_MB goes from 4096 to 8192.
  • API: songMomentsBodySchema accepts optional tempo_hint (number, 30–300) and passes it to Modal. isMeasuredSong types the new fields as optional and checks each one's top-level shape when it is present. Route JSDoc lists the new fields.
  • modal/README.md: field table, the tempo-hint rule, and the calibration caveat.

Decisions

  • One loudness figure: loudness_lufs is replaced by the stereo ebur128 value, and lufs equals it. Without a file (or if ffmpeg fails), the mono pyloudnorm estimate stands. true_peak_dbtp, loudness_range_lu and broadcast_ready are null in that case; we don't fall back to the sample peak.
  • measured_with is <base version>+extended-2026-11-04, so stored results can be rerun when either part changes.
  • tempo_bpm is the reconciled tempo. tempo_note says whether it came from the tracker, was reconciled to the hint (t, t/2, 2t, 3t/4, 4t/3, 2t/3 or 3t/2 within 8 %), or the hint won.
  • mmss is kept inside mute_drops and edit_points (where it is part of the object). It is not added to sections/moments, whose shape is unchanged. The prototype's file field is dropped because the endpoint takes a URL.
  • Strict JSON: NaN and infinity become null, and numpy scalars become plain Python values, so the FastAPI response never fails to serialize.
  • Calibration caveat: the fingerprint formulas and the band reference split were calibrated on two songs. A corpus pass should replace them (and bump EXTENDED_VERSION).

Cost

The extended pass is the slow part. On a laptop it adds about 25 s to a four-minute song, about 70 % of it the harmonic/percussive split behind fingerprint.organic_electronic. A fifteen-minute file peaks near 5.5 GB, hence 8 GB. If the extra time hurts the 300 s route budget, run that split at 22 kHz or on the magnitude spectrogram; the fingerprint would then need recalibrating.

Verification

  • Parity: on the same synthetic stereo file with --tempo-hint 86, the prototype and this port give identical values for every field. The only differences are measured_with, the dropped file, and tempo_note formatting ("86" not "86.0").
  • Python, 24 passed (5 existing + 19 new), Python 3.11: uv run --no-project --with librosa==0.10.2.post1 --with pyloudnorm==0.1.1 --with pytest python -I -m pytest modal/test_song_moments.py modal/test_song_measure.py -q. The new tests use synthetic signals:
    • a 120 BPM click track with a silent two-beat gap is found at 16.0 s
    • a chord in A gives tonic A
    • fingerprint values stay within 0–1
    • band shares sum to about 100
    • a 117.5 tracker read with hint 86 reconciles to 88.1 (3/4)
    • ebur128 parsing, edit points, the no-file path, and strict JSON
    • measure_file with ffmpeg
  • npx vitest run lib/songMoments: 46 passed (9 files). eslint (--max-warnings=0), prettier and tsc -p tsconfig.build.json are clean.

Not deployed. Modal deploys on merge to main. The docs site's OpenAPI entry for /api/songs/moments will need the new fields and tempo_hint.

🤖 Generated with Claude Code

Summary by cubic

Extends the CPU song-moments measurement with the full set of numbers a song report draws: loudness, stereo width, spectral bands, fingerprint, key, drops, and edit points. No language model is involved. Stacked on #1225 (codex/song-moments); review and merge that first.

New features

  • modal/song_measure.py adds measure_extended(), returning the base measurement plus lufs, true_peak_dbtp, sample_peak_dbfs, crest_factor_db, loudness_range_lu, broadcast_ready, stereo, spectral_centroid_hz, rolloff95_hz, band_profile, fingerprint, fingerprint_inputs, key, time_signature, valence, arousal, mute_drops, chord_loop, edit_points, tempo_note, and measured_with.
  • measure_file gains a tempo_hint argument and returns the extended measurement; measure_song is unchanged.
  • The Modal endpoint accepts an optional tempo_hint (30–300 BPM), decodes stereo at 44.1 kHz, runs ffmpeg ebur128 on the downloaded file before deleting it, and returns the extended fields. Memory goes from 4096 to 8192 MB for fifteen-minute files.
  • The API body schema accepts tempo_hint and passes it to Modal; the response type guard checks the new optional fields' top-level shapes when present.

Notes

  • loudness_lufs is replaced by the stereo ebur128 value and lufs equals it. Without a file (or if ffmpeg fails) the mono pyloudnorm estimate stands, and true_peak_dbtp, loudness_range_lu, and broadcast_ready are null.
  • measured_with becomes <base version>+extended-2026-11-04 so stored results can be rerun when either part changes.
  • The fingerprint formulas and band reference split were calibrated on two songs; a corpus pass should replace them and bump EXTENDED_VERSION.
  • The extended pass adds about 25 s to a four-minute song on a laptop (mostly the harmonic/percussive split behind fingerprint.organic_electronic); a fifteen-minute file peaks near 5.5 GB. The tempo hint reconciles the beat tracker's reading when a half/double/3:4 multiple fits within 8 %.
  • Python tests (24 passed) cover the new helpers with synthetic signals; npx vitest run lib/songMoments (46 passed) covers the type guard and schema.
  • Not deployed; Modal deploys on merge to main. The docs site's OpenAPI entry for /api/songs/moments needs the new fields and tempo_hint.