Researchers from Columbia University and Databricks Mosaic AI have conducted a comparative study of full finetuning and Low-Rank Adaptation (LoRA), a parameter-efficient finetuning method, in large language models (LLMs). The efficient finetuning of LLMs, which can contain billions of parameters, is an ongoing challenge due to the substantial GPU memory required. This makes the process expensive and resource-intensive, a significant issue since these models must adapt to new tasks while retaining old capabilities in applications such as natural language processing and artificial intelligence.
The study compared the performance of LoRA and full finetuning – adjusting all model parameters, which is computationally expensive – across programming and mathematics domains using instruction finetuning and pretraining with unstructured tokens. The researchers found that LoRA generally underperformed compared to full finetuning in programming and mathematics tasks, with full finetuning achieving a higher HumanEval score in programming and a higher GSM8K score in mathematics. However, LoRA was shown to provide beneficial regularisation, helping maintain the base model’s performance on tasks outside the target domain, surpassing common techniques such as weight decay and dropout.
Additionally, a detailed analysis revealed that full finetuning resulted in weight perturbations that ranked 10 to 100 times greater than those used in LoRA configurations. This discrepancy in rank explains some of the performance gaps observed. Broader analyses also showed that LoRA’s lower rank perturbations contribute to maintaining more diverse output generations than full finetuning, leading to fewer but potentially more creative solutions.
Though LoRA proved less effective in terms of accuracy and efficiency, it offered advantages in regularisation and memory efficiency. The researchers suggested optimising hyperparameters and understanding the trade-offs between learning and forgetting could improve LoRA’s application versatility. Additionally, despite full finetuning’s better performance, LoRA’s ability to maintain the base model’s capabilities and generate diverse outputs makes it valuable in few contexts, providing important insights into striking a balance between performance and computational efficiency when finetuning LLMs.
