Riffusion

Generated spectrogram from the prompt "bossa nova with electric guitar" (top), and the resulting audio after conversion (bottom)

Riffusion is a neural network, designed by Seth Forsgren and Hayk Martiros, that generates music using images of sound rather than audio.¹

The resulting music has been described as "de otro mundo" (otherworldly),² although unlikely to replace man-made music.² The model was made available on December 15, 2022, with the code also freely available on GitHub.³

The first version of Riffusion was created as a fine-tuning of Stable Diffusion, an existing open-source model for generating images from text prompts, on spectrograms,¹ resulting in a model which used text prompts to generate image files which could then be put through an inverse Fourier transform and converted into audio files.³ While these files were only several seconds long, the model could also use latent space between outputs to interpolate different files together¹⁴ (using the img2img capabilities of SD).⁵ It was one of many models derived from Stable Diffusion.⁵

In December 2022, Mubert⁶ similarly used Stable Diffusion to turn descriptive text into music loops. In January 2023, Google published a paper on their own text-to-music generator called MusicLM.⁷⁸

Forsgren and Martiros formed a startup, also called Riffusion, and raised $4 million in venture capital funding in October 2023.⁹¹⁰