Colloquium 2024 Debugging and Optimization of PyTorch Models

Deep learning models are often viewed as uninterpretable "black boxes". As researchers, we often extend this thinking to the memory and compute utilization of such models. Using PyTorch Profiler, we can identify model bugs and bottlenecks to understand how to improve model performance from an efficiency perspective. This will improve training scaling and allow completion of large hyperparameter optimizations more efficiently. Here we will dicuss the usage of PyTorch Profiler, including some case studies of real training examples, and discuss possible optimizations based on profiler results.