11 CUTLASS
This chapter covers
- Why CUTLASS lets us fuse GEMM workflows beyond cuBLAS
- Hierarchical tiling that maps grids → threadblocks → warps
- Extending kernels with CuTe layout and iterator abstractions
- Tuning tile shapes, pipeline stages, and epilogues for throughput
- Comparing architectures and precision choices from Ampere to Blackwell
- Profiling and scaling CUTLASS kernels, including multi-GPU execution
CUTLASS (CUDA Templates for Linear Algebra Subroutines and Solvers) matters when the default GEMM path is almost right but not quite enough. Once a workload needs fused epilogues, custom tile shapes, architecture-specific scheduling, or multi-GPU-aware kernels, the convenience of a black-box library starts to involve a tradeoff with the control needed for performance work. This chapter puts that tradeoff in concrete terms by showing where CUTLASS sits between using cuBLAS as is and writing every kernel detail by hand.
Along the way, we’ll cover how a GEMM is decomposed across the GPU hierarchy, how CUTLASS 3.x assembles kernels from reusable pieces, how precision and architecture choices affect the implementation, and where profiling fits into the workflow when you need to justify using a custom kernel at all.
11.1 Why CUTLASS matters
So far, cuBLAS has been the default answer for GEMM-heavy workloads. We hand it our matrices, and it returns fast, reliable results. For a large class of problems, that is exactly the right choice.