Information Theory and Generative Modeling

Catalog description

A graduate-level introduction to information theory, data compression, and generative modeling. An introduction to information measures: entropy, mutual information, relative entropy, differential entropy, and their properties. Lossless compression and its connection to prediction and generative modeling. The Minimum Description Length (MDL) principle in model selection. Practical lossless compression using arithmetic coding. The rate-distortion theorem and its connection to lossy compression standards such as JPEG, mp3, and AAC as well as generative modeling techniques such as autoencoders and variational inference. The Nonlinear Transform Coding framework. Practical methods for lossy compression such as Trellis-Coded Quantization (TCQ) and entropy-constrained dithered quantization. ECE 5620 is intended for M.Eng. students and advanced undergraduates with assignments focused on the development of practical data compression algorithms.