Large Language Models (LLMs) signify a major stride in artificial intelligence with their strong natural language understanding and generation capabilities. They can perform plenty of tasks ranging from powering virtual assistants to generating substantial content and conducting profound data analysis. Nevertheless, one obstacle LLMs face is generating factually correct responses. Often, due to the wide array of data they digest, they produce misleading or inaccurate information.
LLMs tend to generate inaccurate or fabricated data, a phenomenon called ‘hallucination.’ This issue generally stems from supervised fine-tuning (SFT) and reinforcement learning (RL) processes that inadvertently prompt these models to generate misleading results. As LLMs are created to respond to diverse user queries, it’s crucial that they deliver accurate information. However, traditional methods like SFT and RL with Human Feedback (RLHF) aimed at improving LLMs’ instruction-following capabilities often prioritize detailed and longer responses leading to increased hallucinations. Furthermore, fine-tuning models with new or unfamiliar data exacerbates the problem, causing them to generate unreliable content.
To address these problems, researchers from the University of Waterloo, Carnegie Mellon University, and Meta AI introduced a novel method called Factuality-Aware Alignment (FLAME). This method improves factual accuracy in LLMs using a blend of factuality-aware SFT and RL with Direct Preference Optimization (DPO). The innovative approach crafts training data that prompts models to produce more factual responses while using specialized reward functions to guide them toward accurate outputs.
FLAME’s approach involves recognizing fact-based instructions requiring factual answers. The model is then fine-tuned using a factuality-aware SFT strategy that prevents the model from being trained on unfamiliar data, which could induce hallucination. The second step uses DPO, which deploys factuality-specific rewards to distinguish between fact-based and non-fact-based instructions. This guides the LLMs to generate more reliable responses, enabling them to follow instructions effectively while reducing the likelihood of hallucination.
The research showed FLAME to considerably enhance LLMs’ factual accuracy, achieving a 5.6-point rise in FActScore compared to standard alignment processes. This was confirmed using the Biography dataset, which appraises the factuality of produced content, and Alpaca Eval, a benchmark that assesses a model’s ability to follow instructions. 805 instruction-following tasks from Alpaca Eval were used to measure the win rate of models using FLAME, showcasing its effectiveness in balancing factuality with the capacity to follow directions.
Thus, FLAME provides a promising solution to the challenges confronting LLMs today. By refining the training and optimization process, it enables LLMs to follow instructions effectively while considerably lowering the risk of hallucination. This makes them more suitable for applications where accuracy is crucial, paving the way for more dependable AI-driven solutions going forward.
