The progress in Language Learning Models (LLMs) has been remarkable, with innovative strategies like Chain-of-Thought and Tree-of-Thoughts augmenting their reasoning capabilities. These advancements are making complex behaviors more accessible through instruction prompting. Reinforcement Learning from Human Feedback (RLHF) is also aligning the capabilities of LLMs more closely with human predilections, further underscoring their visible progression.
In recent research, scientists from Meta, the Georgia Institute of Technology, StabilityAI, and UC Berkeley have investigated the extent to which various Reinforcement Learning (RL) algorithms can boost the reasoning capabilities of LLMs across assorted reward schemes, model dimensions, and initializations. Of the tested algorithms, Expert Iteration (EI) continually outperforms others and shows competitive sample efficiency. RL fine-tuning has been identified as significant in reducing the performance gap between pre-trained and supervised fine-tuned LLMs. The exploration is cited as a critical factor affecting RL fine-tuning efficacy within LLMs.
LLMs can now deal with complex tasks, backed by advanced tools like CoT and Tree of Thought techniques. Linking LLMs with planning algorithms and instruments further augments their capacity to reason. Despite much research into the use of RL for improving LLMs, experts are yet to grasp the understanding of the most impacting factors fully.
Reasoning tasks are perceived as RL problems for LLMs during investigations. This process involves scrutinizing various RL algorithms’ success and sample complexity while fine-tuning the LLMs. The goal of each algorithm is to maximize the expected future return of a student policy on designated tasks. Trials have been conducted on reasoning tasks with these algorithms, showing their efficacy in bolstering LLM achievements.
In these investigations, GSM8K and SVAMP datasets are utilized to evaluate different models using assorted metrics. Initially, Supervised Fine-Tuning (SFT) statistics are used, succeeded by experiments without SFT statistics. EI consistently demonstrates superior performance, marking a significant improvement over the baseline. The findings reveal that RL fine-tuning, especially EI, gives a better generalization and diversity in solution paths than static SFT fine-tuning.
The conclusions drawn from these studies underscore EI’s superiority over other RL algorithms in reasoning tasks. Quick convergence is observed with EI and PPO without supervised fine-tuning, with negligible benefits drawn from additional guidance or denser rewards. RL fine-tuning shows improvement in single- and multi-step accuracy by leveraging dynamic synthetic data generation. Continued advancements in prompting strategies and model exploration are deemed vital for bolstering Language Model reasoning abilities.
