Add Music Flamingo

Merged
LA
8mo ago
+2455-73

This PR adds support for Music Flamingo, NVIDIA's open large audio-language model designed for deep music understanding and reasoning.

Built on Audio Flamingo 3, Music Flamingo specializes in music analysis and long-form audio reasoning, extending maximum audio support to 20 minutes (vs. 10 minutes in AF3) via Rotary Time Embeddings (RoTE). It also adds a more comprehensive music-focused system prompt, introduces audio boundary tokens (<|sound_bos|>, <|sound_eos|>) for better audio sequence modeling.

It introduces:

  • MusicFlamingoForConditionalGeneration model class
  • MusicFlamingoProcessor for preprocessing text + audio
  • MusicFlamingoEncoder with RoTE for extended temporal modeling
  • Configuration, modeling, and processing utilities
  • Tests and docs

Music Flamingo can be loaded directly from the Hugging Face Hub:

from transformers import MusicFlamingoForConditionalGeneration, AutoProcessor

processor = AutoProcessor.from_pretrained("nvidia/music-flamingo-2601-hf")
model = MusicFlamingoForConditionalGeneration.from_pretrained("nvidia/music-flamingo-2601-hf", device_map="auto")

conversation = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "Describe this track in detail - genre, tempo, key, instruments, and mood."},
        {"type": "audio", "path": "song.mp3"}
    ]
}]

inputs = processor.apply_chat_template(conversation, tokenize=True, add_generation_prompt=True, return_dict=True).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500)
print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True))
# Example: "This energetic Eurodance track at 150 BPM in E major features bright synth arpeggios..."