Discrete Generative Models

Catalog description

This course is a rigorous, mathematically-focused introduction to the theory and practice of discrete generative models. We begin with a rapid overview of transformer architectures and autoregressive modeling, using them as a familiar entry point to sequence generation and large language models. Building on this foundation, the course develops the subject from first principles, introducing the probabilistic and algorithmic underpinnings of Markov chains, stochastic processes, and stochastic differential equations. From there, we cover modern frameworks that power state-of-the-art generative methods, including score matching, discrete diffusion, flow matching, optimal transport, and Schrödinger bridges. Advanced topics will include Stein operators and kernelized Stein discrepancies, variational principles such as the Donsker-Varadhan representation and ELBO, functional inequalities including Poincaré and log-Sobolev bounds, and nonparametric priors such as Dirichlet and Pitman-Yor processes. Each framework will be studied both theoretically and practically. On the theoretical side, we will derive key theorems, convergence results, and variational characterizations, and prove guarantees for mixing, expressivity, and invariance. On the practical side, students will design and implement algorithms from scratch, experiment with coding instantiations of each framework, and analyze their behavior on real discrete data. Although the course is theory driven, applications will focus on discrete biological sequence domains, including DNA, RNA, peptides, proteins, single cell data, and ‘omics datasets, to demonstrate how discrete generative models provide a principled foundation for sequence design, modeling, and analysis. By the end of the semester, students will be able to (i) derive and analyze generative processes on discrete state spaces, (ii) implement and experiment with…