Artificial Intelligence Models & Data Resources

Catalog description

This AI evals course covers the role of data in refining and evaluating AI systems, emphasizing AI evals, evaluation metrics, and evaluation techniques used to assess generative AI and modern AI applications. Students explore how evaluation criteria, objective criteria, and clear metrics are used to measure AI outputs and model performance, particularly in complex AI systems that differ from traditional software testing. The course examines evaluating AI systems through a systematic approach, including llm evaluation, evaluating AI agents, and the use of representative data and ground truth. Topics include error analysis, identifying failure modes, handling edge cases, and understanding how evaluation supports continuous improvement and user trust. Through hands-on activities and real world examples, students develop practical skills to implement effective AI evals and support continuous evaluation of evolving AI products. The course prepares data scientists and technical practitioners to apply AI evaluation methods aligned with business requirements and reliable system performance. (Prerequisites: ARIN102 and CSCI360)