Deep neural networks (DNNs) have found widespread success across various fields. This success can be attributed to first-order optimizers such as stochastic gradient descent with momentum (SGDM) and AdamW. However, these methods encounter challenges in efficiently training large-scale models. As an alternative, second-order optimizers like K-FAC, Shampoo, AdaBK, and Sophia have demonstrated superior convergence properties,…
