Researchers from East China University of Science and Technology and Peking University have conducted a survey exploring the use of Retrieval-Augmented Language Models (RALMs) within the field of Natural Language Processing (NLP). Traditional methods used in this field, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short Term Memory (LSTM), have seen advancements through the incorporation of transformer architecture and large language models (LLMs) like Generative Pretrained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) families.
Despite their success, LLMs face challenges such as hallucination (the forming of content that is not present in the source) or the necessity for a domain-specific knowledge base. RALMs, such as Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU) were created to meet these challenges, enhancing NLP tasks by incorporating external information and refining its output. The researchers revealed that this has led to the expansion of applications in areas of translation, dialogue generation, and knowledge-intensive applications.
RALMs function by using retrieved information to refine the output of language models. This is categorized into sequential single interaction, sequential multiple interaction, and parallel interaction. Sequential single interaction sees retrievers identifying and distributing relevant documents to the language model for output prediction. Sequential multiple interactions allow for the model to undergo an iterative refinement process, whereas parallel interactions allow both retrievers and language models to operate independently and interpolate their outputs.
Enhancements of RALMs can come in the form of improving retrievers, language models, or the overall architecture of the system. Improvements in retriever function focus on optimizing timing and utilizing quality control to ensure the correct documents are selected for use. Improvements to the language model could involve pre-generation retrieval processing and structural model optimization. As for the enhancement of the overall system, actions such as end-to-end training and the implementation of intermediate modules can be undertaken.
RAG and RAU are specialized RALMs, created for natural language generation and understanding respectively. RAG is dedicated to enhancing tasks such as text summarization and machine translation, while RAU is designed to carry out understanding tasks like question-answering and reasoning.
With their versatility, RALMs have been used in a variety of NLP tasks. They have been incorporated into machine translation, dialogue generation, text summarization, code summarization, question answering, and the completion of knowledge graphs. In all these tasks, RALMs have shown adaptability and efficiency, demonstrating their broad potential in natural language understanding and generation.
In conclusion, the survey elucidated how RALMs such as RAG and RAU are advancing NLP by coalescing external data retrieval with large language models, thus enhancing their performance across multiple tasks. Research efforts have been put into the refinement of the retrieval-augmented paradigm to optimize retriever-language model interaction. As NLP evolves, RALMs offer a promising direction for future studies and development in order to improve our aptitude in computational language understanding.
