PyTorch has introduced TK-GEMM, an enhanced Triton FP8 GEMM (General Matrix-Matrix Multiply) kernel, designed to expedite FP8 inference for large language models (LLMs) such as Llama3. This new development responds to the struggle faced in standard PyTorch execution, where multiple kernels are launched on the GPU for each operation in LLMs, typically leading to inefficient…