Scaling laws in artificial intelligence are fundamental in the development of Large Language Models (LLMs). These laws play the role of a director, coordinating the growth of models while revealing patterns of development that go beyond mere computation. With every new step, the models become more nuanced, accurately deciphering the complexities of human expression. Scaling laws don’t just govern language but offer infinite potential for it, located at the intersection of understanding and creation.
Despite their significance, there’s a disconnect between current scaling studies and how LLMs are truly trained and evaluated. Training such models is pricey and they are often over-trained to minimize inference costs and evaluate their performance on downstream tasks. To train top-tier models implies employing a blend of algorithmic approaches and training data. In this process, reliable extrapolation is frequently employed in the final training phase, and is a standard practice for state-of-the-art language models like Chinchilla 70B, PaLM 540B, and GPT-4.
In an effort to understand when scaling is predictable in the over-trained phase, researchers from various universities experimented with a testbed of 104 models having parameter counts ranging from 0.011B to 6.9B, and trained these with different numbers of tokens on three distinct datasets: RedPajama, C4, and Refined Web. The findings from this experiment were then used to predict the validation loss of two models – one with 1.4B parameters and 900B tokens run, and the other with 6.9B parameters and 138B tokens run.
The research revealed that scaling laws, when applied to smaller models that are trained closer to the compute-optimal, are effective in predicting the performance of larger models that are subject to more expansive over-training. However, predicting errors on individual tasks remains challenging, leading the researchers to reliably forecast aggregate performance based on a model’s perplexity in comparison to models trained on the same dataset.
Another significant observation was consistency in power laws related to the models’ reducible loss when a set of model configurations maintained a constant ratio of training tokens to parameters. It was also found that the scaling exponent is unaffected by an increase in the ratio of tokens to parameters, although the scalar experiences changes.
To determine the degree of over-training, token multipliers were applied to well-known models, with the range spanning from 5 to 640. The research also revealed exponential decay of average top-1 error alongside a decrease in C4 evaluation loss.
To sum it up, the study made significant strides in predicting the downstream average task performance of more expensive runs using smaller-scale proxies and gave important insights into scaling in the over-trained regime. However, the work is unfinished, and future research could focus on hyperparameters incorporation and developing an analytical theory to explain why scaling fails in some cases.
Feel free to look into the paper and Github for more information. All credit for the research goes to the researchers of the project. Also, consider following us on Twitter for regular updates and join our Telegram, Discord channels, and LinkedIn group for engaging discussions. Don’t forget our subreddit if you are a Machine Learning enthusiast. And lastly, if you appreciate our work, do sign up for our newsletter to stay in the loop.
