GPU Cluster Programming. 3 credits, 3 contact hours

Catalog description

Prerequisites: CS 610 and CS 630 and familiarity with Linux Programming. This project-oriented course equips students with advanced problem-solving skills on CUDA-capable Linux clusters by integrating two powerful programming models. Students harness MPI to program clusters using the MIMD (Multiple Instruction Multiple Data) model, mastering various communication techniques. Simultaneously, they delve into CUDA programming to exploit thousands of GPU cores under the SIMD (Single Instruction Multiple Data) model. By adopting the SPMD (Single Program Multiple Data) paradigm, students learn to synergize MPI and CUDA to tackle large-scale challenges such as training advanced neural networks for generative and agentic AI, as well as performing complex inference with latent search. The course also covers essential CUDA methodologies, including convolution, histogram, reduction, prefix sum (scan), merge, radix sorting, and graph traversal, culminating in a comprehensive, hands-on project that addresses a real-world problem on a CUDA-enabled Linux cluster.