Large Language Models (LLMs) have transformed the field of Natural Language Processing (NLP), specifically in Question Answering (QA) tasks. However, their utility is often hampered by the generation of incorrect or unverified responses, a phenomenon known as hallucination. Despite the development of advanced models like GPT-4, issues remain in accurately answering questions related to changing facts or less popular entities. Retrieval-Augmented Generation (RAG), which uses external data to inform answer generation, holds promise in addressing these challenges. However, it comes with its hurdles, including selecting relevant information, reducing query times, and effectively using information for complex inquiries.
In response to these challenges, researchers from Meta Reality Labs, FAIR, Meta, HKUST, and HKUST (GZ) have introduced the Comprehensive benchmark for RAG (CRAG). This benchmark incorporates five essential features: realism, richness, insightfulness, reliability, and longevity. It includes 4,409 different question-answer pairs from five domains, from straightforward fact-based questions to seven kinds of complex inquiries. These questions, manually verified and paraphrased, cover varied entity popularity and time spans, which enable insights into model performance across a wide range of conditions. CRAG simulates realistic noise by providing mock APIs for web-page retrievals and knowledge graphs containing 2.6 million entities. The benchmark tests the web retrieval, structured querying, and summarization abilities of RAG solutions via three different tasks.
The findings indicate the efficacy of the CRAG benchmark. While sophisticated models such as GPT-4 only attain about 34% accuracy on CRAG, integrating a simple RAG improves accuracy to 44%. Even so, state-of-the-art commercial RAG solutions can answer merely 63% of questions without hallucination, primarily struggling with more dynamic, intricate, or less popular facts. These results emphasize the suitable difficulty level of CRAG and its ability to reveal meaningful insights from diverse data. Further, it exposes the gaps to bridge in designing fully trustworthy QA systems, positioning it as a useful benchmark to stimulate further progress in NLP research.
The researchers express their ongoing commitment to enhancing and expanding CRAG to include multilingual questions, multi-modal inputs, and multi-turn conversations. They see CRAG as playing a leading role in advancing RAG research, adapting to emerging challenges, and meeting new research needs in the fast-evolving field of NLP. This benchmark provides a sturdy base for improving reliable, grounded language generation abilities. Given its usefulness, CRAG is poised to be an essential tool for QA system development, pushing the boundaries of current capabilities and enabling progress in tackling the hallucination challenge.
Regarding this project’s findings, the researchers urge the wider scientific community to further explore the potential of RAG and continuous evaluation through benchmarks like CRAG.
