Artificial intelligence has witnessed significant progress with the creation of large language models (LLMs). Techniques such as reinforcement learning from human feedback (RLHF) have dramatically improved AI’s ability to execute various tasks. However, generating new content based purely on human feedback presents a challenge. Optimizing the LLM’s learning process from human feedback is an essential task. Here, models respond to prompts, and human raters exhibit their preferences, which helps fine-tune the model’s responses to align more closely with human preferences.
Currently, LLM training methodologies involve passive exploration, where responses are generated based on fixed prompts, without active attempts to enhance the learning from feedback. Various exploration schemes, such as Thompson sampling, Boltzmann Exploration, and Infomax, have been utilized, though these require numerous human interactions to achieve considerable improvements.
A team of researchers from Google Deepmind and Stanford University have proposed a new approach using double Thompson sampling and Epistemic Neural Network (ENN) for query generation. This method enables the model to actively pursue the most instructive feedback, reducing the required queries for high performance. ENN provides uncertainty estimates directing the exploration process, letting the model choose which queries to present for feedback more logically.
The experiment involved modeling agents generating responses to 32 prompts. Feedback from these responses was utilized to refine their reward models. These agents explored the response-space by selecting the most informative pairs from a pool of 100 candidates.
Outcome of the study showed double Thompson sampling was more efficient than Boltzmann exploration and infomax due to utilization of uncertainty estimates for better query selection. This process accelerates learning, dramatically reducing the human feedback volume required.
The research highlights the potential of efficient exploration in overcoming the boundaries of traditional training techniques. By exploiting advanced exploration algorithms and uncertainty estimates, this approach promises to speed up innovation in LLMs. This highlights the necessity of optimizing the learning process for AI advancement.
The research was conducted by a joint team from Google’s DeepMind and Stanford University, both lauded for their extensive contributions to the AI research field. The success of their efficient exploration strategy opens up new opportunities for the faster and more effective development of AI models.
Nikhil, an AI/ML enthusiast, intern consultant at Marktechpost, is currently researching applications in fields like biomaterials and biomedical science. Balancing a background in Material Science, Nikhil’s work represents the incredible potential that lies in the intersection of AI with diverse fields of study.
