Add Music Flamingo
Merged
LA
8mo ago+2455+2455+0123456789012345678901234567890123456789-73-73-01234567890123456789
This PR adds support for Music Flamingo, NVIDIA's open large audio-language model designed for deep music understanding and reasoning.
- Paper: Music Flamingo: Scaling Music Understanding in Audio Language Models
- Model Weights: nvidia/music-flamingo-2601-hf
- Demo: Music Flamingo Demo
- Project Page: NVIDIA Research
Built on Audio Flamingo 3, Music Flamingo specializes in music analysis and long-form audio reasoning, extending maximum audio support to 20 minutes (vs. 10 minutes in AF3) via Rotary Time Embeddings (RoTE). It also adds a more comprehensive music-focused system prompt, introduces audio boundary tokens (<|sound_bos|>, <|sound_eos|>) for better audio sequence modeling.
It introduces:
MusicFlamingoForConditionalGenerationmodel classMusicFlamingoProcessorfor preprocessing text + audioMusicFlamingoEncoderwith RoTE for extended temporal modeling- Configuration, modeling, and processing utilities
- Tests and docs
Music Flamingo can be loaded directly from the Hugging Face Hub:
from transformers import MusicFlamingoForConditionalGeneration, AutoProcessor
processor = AutoProcessor.from_pretrained("nvidia/music-flamingo-2601-hf")
model = MusicFlamingoForConditionalGeneration.from_pretrained("nvidia/music-flamingo-2601-hf", device_map="auto")
conversation = [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this track in detail - genre, tempo, key, instruments, and mood."},
{"type": "audio", "path": "song.mp3"}
]
}]
inputs = processor.apply_chat_template(conversation, tokenize=True, add_generation_prompt=True, return_dict=True).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500)
print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True))
# Example: "This energetic Eurodance track at 150 BPM in E major features bright synth arpeggios..."