Introduction to GPU Cluster Programming. 3 credits, 3 contact hours

Catalog description

Prerequisites: CS 288 , CS 332 and CS 350 , each with a grade C or better. Students learn problem-solving by working with a cluster of CUDA-capable Linux computers. They explore an overarching programming paradigm that integrates two distinct architectural models to tackle complex challenges, such as training generative or agentic AI models using convolution neural networks and performing inference with latent search. Through the Message Passing Interface (MPI), students learn to program clusters of Linux computers, embodying the Multiple Instruction Multiple Data (MIMD) architectural model. Simultaneously, they delve into Compute Unified Device Architecture (CUDA) to program thousands of GPU cores, following the Single Instruction Multiple Data (SIMD) architectural model. By employing the Single Program Multiple Data (SPMD) programming paradigm, students combine MPI and CUDA to address large-scale problems on CUDA-capable Linux clusters effectively. The topics on MPI entail mainly communication mechanisms between many computers, which are point to point, collective, and one-sided. The topics on CUDA entail primarily methodologies to solve essential computing problems, including convolution, histogram, reduction, prefix sum (scan), merge, radix sorting, graph traversal, and convolution neural networks. The course is project-oriented and as such at the end of semester, students will be able to solve a difficult problem using a cluster of CUDA-capable computers.