GPU Computing for AI

Graphics Processing Units (GPUs) have become the dominant compute platform for artificial intelligence workloads. Unlike CPUs optimized for sequential, latency-sensitive tasks, GPUs are massively parallel processors designed for throughput computing. This chapter provides a comprehensive technical treatment of GPU architecture, programming models, performance optimization, and deployment considerations for AI workloads. We cover NVIDIA, AMD, and emerging GPU alternatives, with deep dives into memory hierarchies, parallelization strategies, and the economics of GPU computing.

GPU Computing for AI refers to the use of Graphics Processing Units to accelerate machine learning training and inference workloads. GPUs achieve their performance advantage through thousands of compute cores executing thousands of threads simultaneously, combined with high-bandwidth memory architectures. The GPU programming model, exemplified by NVIDIA CUDA and AMD ROCm, allows developers to express data-parallel algorithms that map efficiently to the GPU massively parallel architecture.

GPU computing is the foundation of modern AI. The transition from CPU-based machine learning to GPU-accelerated deep learning in the early 2010s enabled the breakthroughs that define today AI landscape. Understanding GPU architecture is essential for selecting the right hardware for training and inference workloads, optimizing performance through memory management and parallelization, estimating costs accurately for infrastructure planning, troubleshooting performance bottlenecks in production, and designing efficient distributed systems that leverage GPU communication.

GPU Market Overview NVIDIA: - H100: Current flagship data-center GPU (80GB HBM3, 3.35 TB/s bandwidth) - H200: Enhanced H100 with additional HBM3E memory (141GB) - B100/B200 (Blackwell): Next-generation architecture announced - RTX 6000 Ada: Workstation GPU for AI development AMD: - MI300X: Data-center accelerator competing with H100 (192GB HBM3, 5.3 TB/s) - ROCm 6.0: Open software stack Intel: - Gaudi 3: AI accelerator - Data Center GPU Max series GPU Architecture Trends: - Increasing HBM capacity and bandwidth - Advanced precision support (FP8, FP4) - Enhanced multi-GPU communication (NVLink, Infinity Fabric) - Higher power density (800W+ per GPU) - Liquid cooling becoming standard Key Metrics: - FLOPS (Floating Point Operations Per Second) - Memory Bandwidth (TB/s) - Interconnect Bandwidth (GB/s) - Precision Support (FP64, FP32, FP16, BF16, FP8, INT8)

• Explain GPU architecture and its advantages for AI workloads
• Analyze GPU memory hierarchies and their impact on model training
• Design efficient GPU kernels for matrix multiplication and attention
• Optimize GPU utilization through batching and parallelism strategies
• Select appropriate GPU configurations for different AI workloads
• Implement distributed training strategies on multi-GPU systems
• Estimate GPU requirements and costs for infrastructure planning
• Evaluate GPU vs TPU vs ASIC trade-offs

• Computer architecture fundamentals
• Understanding of AI/ML training and inference
• Basic knowledge of parallel computing concepts
• Familiarity with matrix multiplication and linear algebra

CPUs are optimized for sequential, latency-sensitive tasks with a few powerful cores, deep cache hierarchies, and sophisticated branch prediction. GPUs are optimized for throughput with thousands of simpler cores, high-bandwidth memory, and massive parallelism. For AI workloads dominated by matrix multiplication and tensor operations, GPUs achieve 10-100x speedup over CPUs. The key architectural difference is that CPUs minimize latency for individual operations while GPUs maximize throughput across thousands of parallel operations.

A GPU die consists of Streaming Multiprocessors (SMs) or Compute Units (CUs), each containing CUDA Cores or Stream Processors, Tensor Cores or Matrix Units, shared memory/L1 cache, and register file. The GPU also includes HBM (High-Bandwidth Memory), memory controllers, interconnect (NVLink/Infinity Fabric), and PCIe/other I/O. The NVIDIA H100 SXM has 132 SMs, each with 128 CUDA cores and 4 Tensor Cores, totaling 16,896 CUDA cores and 528 Tensor Cores, with 80GB HBM3 and 3.35 TB/s bandwidth.

GPU memory hierarchy flows from CPU system memory through PCIe (32-64 GB/s) to GPU HBM (3-5 TB/s), L2 cache, L1 cache/shared memory, register file, and compute units. Memory management for AI includes parameters (params × precision bytes), activations (batch × sequence × hidden), KV cache (batch × sequence × heads × head_dim × 2 × precision), and optimizer states (2 × parameters for Adam). Parallelization strategies include data parallelism (model replicated, data split), model parallelism (layers partitioned), tensor parallelism (tensors partitioned), and pipeline parallelism (micro-batches through stages).

Model Memory Requirements: MEMORY = PARAMETERS + ACTIVATIONS + KV CACHE + OPTIMIZER STATES + OVERHEAD PARAMETERS = (PARAMS × PRECISION_BYTES) ACTIVATIONS = BATCH × SEQUENCE_LENGTH × HIDDEN_SIZE KV CACHE = BATCH × SEQUENCE × HEADS × HEAD_DIM × 2 × PRECISION_BYTES OPTIMIZER STATES = 2 × PARAMETERS (for Adam) Example: 70B Parameter Model - Parameters (FP16): 70B × 2 bytes = 140GB - Single H100 (80GB) is not enough - Need multi-GPU with model parallelism - 8x H100 provides 640GB total (80GB each) Precision and Memory: - FP32: 4 bytes per parameter - FP16/BF16: 2 bytes per parameter - FP8: 1 byte per parameter - INT8: 1 byte per parameter - INT4: 0.5 bytes per parameter Quantization reduces memory requirements significantly: - FP16 to INT8: 50% memory reduction - FP16 to INT4: 75% memory reduction

Performance Bottlenecks: Compute-Bound: - Matrix multiplication dominates - Tensor cores fully utilized - Solution: Increase batch size, optimize kernels Memory-Bound: - Memory bandwidth saturation - Insufficient HBM capacity - Solution: Reduce precision, offload to CPU Communication-Bound: - Multi-GPU synchronization overhead - Network latency dominates - Solution: Use NVLink, InfiniBand, optimize topology Optimization Techniques: - Kernel Fusion: Combine operations to reduce memory traffic - Reduced Precision: Use FP16/BF16 for training, FP8/INT8 for inference - Memory Pooling: Reuse GPU memory allocations - Asynchronous Transfers: Overlap data transfer with computation - Batching: Process multiple inputs simultaneously GPU Utilization Metrics: - Compute utilization: % of peak FLOPS - Memory utilization: % of peak bandwidth - SM occupancy: % of available threads running

• Build GPU cluster operations in GCCs for global enterprises
• Develop GPU infrastructure engineering and optimization
• Create GPU scheduling and resource allocation systems
• Establish GPU FinOps and cost optimization operations
• Build GPU performance monitoring and optimization
• Develop distributed training operations on GPU clusters
• Create GPU security and access management
• Build cross-market GPU operations for global AI training