Socio-technical Evaluations of Generative AI

Catalog description

This course aims to introduce students to the growing number of evaluation metrics, measures, and methods proposed for assessing the capabilities and safety risks of Generative AI systems when they are deployed in society in ways that impact human lives and wellbeing. The course will also provide a methodological framework for designing and evaluating existing evaluation metrics, methods, and approaches. We formalize the necessary characteristics of evaluation benchmarks and automated methods to ensure that they capture the benefits and risks of GenAI systems in deployment. We focus on four key criteria: Validity, Reliability, Feasibility, and Usability. Through course projects, students will be prompted to design and/evaluate methods for evaluating the risks and capabilities of GenAI applications. Prerequisites: 15-122 Min. grade C and ( 15-151 Min. grade C or 21-127 Min. grade C) and (36-217 Min. grade C or 15-259 Min. grade C or 15-359 Min. grade C or 36-225 Min. grade C) and (10-723 Min. grade C or 10-623 Min. grade C or 10-423 Min. grade C or 10-315 Min. grade C or 10-601 Min. grade C or 10-301 Min. grade C or 10-715 Min. grade C or 10-701 Min. grade C)