Large Language Models (LLMs) like GPT-3.5 Turbo and Mistral 7B often struggle to maintain accuracy while retrieving information from the middle of long input contexts, a phenomenon referred to as “lost-in-the-middle”. This complication significantly hampers their effectiveness in tasks requiring the processing and reasoning over long passages, such as multi-document question answering (MDQA) and flexible length question answering (FLenQA).
Current improvement methods for LLMs typically involve finetuning with real-world datasets, but these often contain outdated or irrelevant information leading to inaccuracies and hallucinations. Traditional datasets like MDQA and FLenQA have displayed that LLMs often operate optimally at the beginning or end of input contexts but exhibit faltering performance with data in the middle.
Researchers from the University of Wisconsin-Madison have proposed a different approach to enhance LLMs’ performance using a synthetic dataset. The dataset comprises numerical key-value retrieval tasks carefully designed to improve LLMs’ ability to handle long contexts without introducing inaccuracies. This methodology helps to avoid the pitfalls of outdated or irrelevant information.
The synthetic dataset consists of simple dictionary key-value retrieval tasks, each involving multiple dictionaries with several keys. The answer part of these tasks is specifically used for finetuning, with other elements masked out to concentrate the model’s learning process. For instance, Mistral 7B’s dataset includes 350 samples, each with 85 dictionaries, producing prompts containing approximately 3900 tokens.
The researchers’ experiments have shown that this approach significantly enhances LLM’s performance in long-context tasks. For example, GPT-3.5 Turbo finetuned on the synthetic dataset displayed a 10.5% improvement on the 20 documents MDQA benchmark at the tenth position. The technique also mitigates the “lost-in-the-middle” issue and reduces primacy bias, resulting in more precise information retrieval across the entire input context. When compared to models finetuned on real-world datasets, the synthetic approach showed superior results in maintaining consistent accuracy across different context positions.
Conclusively, this research introduces a novel approach to finetuning LLMs using synthetic data, significantly improving their efficacy in long-context environments. The method provides substantial enhancements over traditional techniques by tackling the “lost-in-the-middle”issue and reducing the primacy bias. The research shows the potential and superiority of synthetic datasets in dealing with the limitations of real-world data, hence paving the path for more reliable and effective LLMs in managing wide-ranging textual information. All credit for this research goes to the team at the University of Wisconsin-Madison.
