Special Topic: Generative AI for Music and Audio

Catalog description

In this seminar class, we will discuss state-of-the-art methods in generative AI for music and general audio (everyday sounds, speech, bioacoustics, etc.), with applications to both generation and understanding. We will examine and compare the two primary families of methods that are used in modern audio generation research: large language models applied to discrete audio tokens, and diffusion models applied to continuous audio representations. With an eye towards offering intuitive controls for music generation, we will also examine classic methods and tasks in music information retrieval such as spectral analysis, synchronization, beat detection, and transcription. Moreover, we will explore emerging topics in generative AI for music and audio such as new architectures, training data attribution, interaction, compression, multimodality, and evaluation. Finally, we will discuss the ethical and societal implications of music generation specifically, and its potential effects on music both economically and culturally. Much of the course activity will center around (1) in-class lectures and demonstrations on small scale datasets, (2) student-led discussions of research papers, and (3) an open-ended research project.