The existing language learning models (LLMs) are advancing yet have been struggling with incorporating new knowledge without forgetting the previous information, a situation termed as “catastrophic forgetting.” The present methods, such as retrieval-augmented generation (RAG), are not very effective in tasks demanding integration of new knowledge from various passages due to encoding each passage in isolation.
To overcome this hurdle, researchers from Ohio State University and Stanford University have introduced the HippoRAG framework, drawing inspiration from neurobiological principles, particularly the hippocampal indexing theory of our brain. Unlike other models that provide LLMs with long-term memory, this unique HippoRAG method creates a network of associations, thus enhancing the model’s ability to navigate and integrate information sourced from multiple passages.
The novelty of HippoRAG lies in its indexing process that employs a graph-based hippocampal index to extract and relate noun phrases and take passages from an instruction-tuned LLM and a retrieval encoder. This method enables HippoRAG to form an extensive web of associations, thus improving its ability to retrieve and integrate knowledge across various passages. For identifying the most relevant passages for answering a query, it uses a personalized PageRank algorithm during retrieval, exhibiting a superior performance compared to other existing RAG methods.
HippoRAG’s methodology consists of two primary phases: offline indexing and online retrieval. In the indexing part, it processes the passages to extract named entities using an instruction-tuned LLM and a retrieval encoder, and with the help of Open Information Extraction (OpenIE), it forms a graph-based hippocampal index. This index captures the relationships between entities and passages, improving its ability to retrieve and integrate information effectively.
In the retrieval phase, it makes use of a 1-shot prompt to extract named entities from a query, and with the retrieval encoder, it encodes them. HippoRAG retrieves the relevant information by identifying query nodes having a high cosine similarity to the query-named entities through its hippocampal index. It also runs the Personalized PageRank (PPR) algorithm over the index for effective pattern completion, thus enhancing knowledge integration performance across various tasks.
When they were put to test on multi-hop question answering benchmarks, including MuSiQue and 2WikiMultiHopQA, the performance of HippoRAG was superior. It outperformed state-of-the-art methods by up to 20%. Especially, HippoRAG’s single-step retrieval showed better performance or comparable to iterative methods like IRCoT, yet was 10-30 times cheaper and 6-13 times faster. This result showcases the potential of the HippoRAG framework to cause a revolutionary change in language modeling and information retrieval.
In short, the innovative HippoRAG framework significantly boosts LLMs. It is both a theoretical advancement and a practical solution to the deep and efficient integration of new knowledge. Inspired by the human brain’s associative memory functions, HippoRAG enhances the model’s ability to retrieve and synthesize information from multiple sources. This research has danced its superior performance in knowledge-intensive NLP tasks, indicating its potential in practical applications requiring continuous knowledge integration.
