Artificial Intelligence (AI) is rapidly advancing, particularly with the creation of large language models (LLMs) with over 70 billion parameters. While these models are crucial for tasks such as translation and content creation, their full potential can only be realized using Reinforcement Learning from Human Feedback (RLHF), a technique that currently faces significant challenges due to the extensive memory requirements of these large models.
Present RLHF methods often involve distributing the LLM over multiple graphics processing units (GPUs) to facilitate training, however, this approach poses problems. High levels of partitioning can lead to memory fragmentation within each GPU, reducing the batch size for effective training and slowing down the entire process. There can also be communication problems between the partitioned units, creating efficiency issues.
Researchers have proposed a RLHF solution named OpenRLHF in response to these issues. It employs two main technologies: Ray, a sophisticated Distributed Task Scheduler, and vLLM, a Distributed Inference Engine. The former smartly allocates the LLM across GPUs without excessive partitioning, optimizing memory use, and speeding up training. In contrast, the latter enhances computation speed by taking advantage of the concurrent processing capabilities of several GPUs.
A comprehensive comparison with an existing framework like DSChat during the training of a large 7B parameter LLaMA2 model displayed significant improvement with OpenRLHF. It achieved quicker training convergence, implying more efficient learning. Moreover, vLLM’s speedy generation capabilities led to a considerable reduction in total training time. Additionally, intelligent scheduling by Ray minimized memory fragmentation, enabling larger batch sizes and quicker training.
To summarize, OpenRLHF’s innovative approach addresses and eliminates the key issues associated with training large LLMs using RLHF. It uses efficient scheduling and accelerated computation to overcome memory restrictions and achieve faster training convergence. This paves the way for further fine-tuning of larger LLMs with human feedback, ushering in a new era of applications in language processing and information interaction that could potentially transform various fields.
The full details of this research can be found in the published paper and GitHub. This research project is solely credited to its researchers. Interested parties are encouraged to follow the team on social media platforms such as Twitter, Telegram Channel, Discord Channel, and LinkedIn group, as well as join the ML SubReddit with over 42k members. They also offer a newsletter for those interested in their work.
