A team from Harvard University and the Kempner Institute at Harvard University have conducted an extensive comparative study on optimization algorithms used in training large-scale language models. The investigation targeted popular algorithms like Adam – an optimizer lauded for its adaptive learning capacity, Stochastic Gradient Descent (SGD) that trades adaptive capabilities for simplicity, Adafactor with its higher memory efficiency, and Lion, a new contender yet to be fully validated in various architectures. This study aims to generate a more precise understanding of each optimizer’s strengths and limitations, with a substantial portion of the research revolving around Signum and Adalayer – simplified iterations of Adam.
The researchers conducted their studies on autoregressive language models across different parameter scales, varying vital hyperparameters to gauge their effect on performance. The language models were built on the C4 dataset and used a T5 tokenizer, with the evaluation centered around validation loss. The investigation dug further into network components like LayerNorm parameters and the final layer to understand their contribution to overall stability.
The outcome of the study indicated that Adam, Adafactor, and Lion showed comparable results in peak performance and stability, while SGD lagged behind. This finding suggests that AI practitioners can choose an optimizer based on factors such as resource availability and implementation easiness without worrying about performance losses. Intriguingly, the study revealed that adaptive strategies played a crucial role mainly in LayerNorm parameters and the last layer, while simpler approaches such as SGD were effective for the rest of the model.
This proposed method allows for a comprehensive analysis of optimizer performance and stability in training language models. By evaluating a range of optimizers across varying settings and model scales, the study provides valuable insights into optimizing large-scale language models. The research signifies progress in AI, potentially making advanced language models more feasible and efficient to train and, in turn, more widely accessible.
