/multimodal-audiocraft

name: audiocraft-audio-generation
description: PyTorch library for audio generation including text-to-music (MusicGen) and text-to-sound (AudioGen). Use when you need to generate music from text descriptions, create sound effects, or perform melody-conditioned music generation.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Multimodal, Audio Generation, Text-to-Music, Text-to-Audio, MusicGen]
dependencies: [audiocraft, torch>=2.0.0, transformers>=4.30.0]

AudioCraft: Audio Generation

Comprehensive guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec.

When to use AudioCraft

**Use AudioCraft when:**

• Need to generate music from text descriptions
• Creating sound effects and environmental audio
• Building music generation applications
• Need melody-conditioned music generation
• Want stereo audio output
• Require controllable music generation with style transfer

**Key features:**

• **MusicGen**: Text-to-music generation with melody conditioning
• **AudioGen**: Text-to-sound effects generation
• **EnCodec**: High-fidelity neural audio codec
• **Multiple model sizes**: Small (300M) to Large (3.3B)
• **Stereo support**: Full stereo audio generation
• **Style conditioning**: MusicGen-Style for reference-based generation

**Use alternatives instead:**

• **Stable Audio**: For longer commercial music generation
• **Bark**: For text-to-speech with music/sound effects
• **Riffusion**: For spectogram-based music generation
• **OpenAI Jukebox**: For raw audio generation with lyrics

Quick start

Installation

# From PyPI
pip install audiocraft

# From GitHub (latest)
pip install git+https://github.com/facebookresearch/audiocraft.git

# Or use HuggingFace Transformers
pip install transformers torch torchaudio

Basic text-to-music (AudioCraft)

import torchaudio
from audiocraft.models import MusicGen

# Load model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Set generation parameters
model.set_generation_params(
    duration=8,  # seconds
    top_k=250,
    temperature=1.0
)

# Generate from text
descriptions = ["happy upbeat electronic dance music with synths"]
wav = model.generate(descriptions)

# Save audio
torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Using HuggingFace Transformers

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import scipy

# Load model and processor
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.to("cuda")

# Generate music
inputs = processor(
    text=["80s pop track with bassy drums and synth"],
    padding=True,
    return_tensors="pt"
).to("cuda")

audio_values = model.generate(
    **inputs,
    do_sample=True,
    guidance_scale=3,
    max_new_tokens=256
)

# Save
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())

Text-to-sound with AudioGen

from audiocraft.models import AudioGen

# Load AudioGen
model = AudioGen.get_pretrained('facebook/audiogen-medium')

model.set_generation_params(duration=5)

# Generate sound effects
descriptions = ["dog barking in a park with birds chirping"]
wav = model.generate(descriptions)

torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)

Core concepts

Architecture overview

AudioCraft Architecture:
┌──────────────────────────────────────────────────────────────┐
│                    Text Encoder (T5)                          │
│                         │                                     │
│                    Text Embeddings                            │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│              Transformer Decoder (LM)                         │
│     Auto-regressively generates audio tokens                  │
│     Using efficient token interleaving patterns               │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│                EnCodec Audio Decoder                          │
│        Converts tokens back to audio waveform                 │
└──────────────────────────────────────────────────────────────┘

Model variants

| Model | Size | Description | Use Case | |-------|------|-------------|----------| | `musicgen-small` | 300M | Text-to-music | Quick generation | | `musicgen-medium` | 1.5B | Text-to-music | Balanced | | `musicgen-large` | 3.3B | Text-to-music | Best quality | | `musicgen-melody` | 1.5B | Text + melody | Melody conditioning | | `musicgen-melody-large` | 3.3B | Text + melody | Best melody | | `musicgen-stereo-*` | Varies | Stereo output | Stereo generation | | `musicgen-style` | 1.5B | Style transfer | Reference-based | | `audiogen-medium` | 1.5B | Text-to-sound | Sound effects |

Generation parameters

| Parameter | Default | Description | |-----------|---------|-------------| | `duration` | 8.0 | Length in seconds (1-120) | | `top_k` | 250 | Top-k sampling | | `top_p` | 0.0 | Nucleus sampling (0 = disabled) | | `temperature` | 1.0 | Sampling temperature | | `cfg_coef` | 3.0 | Classifier-free guidance |

MusicGen usage

Text-to-music generation

from audiocraft.models import MusicGen
import torchaudio

model = MusicGen.get_pretrained('facebook/musicgen-medium')

# Configure generation
model.set_generation_params(
    duration=30,          # Up to 30 seconds
    top_k=250,            # Sampling diversity
    top_p=0.0,            # 0 = use top_k only
    temperature=1.0,      # Creativity (higher = more varied)
    cfg_coef=3.0          # Text adherence (higher = stricter)
)

# Generate mult