The internet is a vast source of knowledge that is continuously expanding and updating. Keeping up with the constant changes and ensuring people have access to the most current and relevant information is a significant challenge in information retrieval. This challenge is compounded by the rise of large language models (LLMs) like chatGPT, used in question-answering systems, which need to process a vast amount of online text data that is subject to minute-by-minute changes.
To combat these issues, researchers are exploring Retrieval Augmented Language Models (RALMs) that source their information from an external document repository. This external corpus, an index of documents like web pages and Wikipedia entries, can be updated to reflect the latest versions of the information it contains. Despite the potential RALMs have shown for answering factual questions, these models still display difficulties with timing, resulting in incorrect responses when dealing with frequently updated information.
Highlighting these issues, researchers at San Jose State University have developed TempRALM, an enhancement for the Atlas model, a leading RALM with few-shot learning extensions. Unlike traditional RALMs, which only consider semantic similarity when retrieving documents, TempRALM also takes into account the temporal relevance of the information in response to the query. The model uses a new temporal retrieval mechanism tested for efficacy, building on the RALM architecture presented by Atlas.
The TempRALM retriever compliments the standard Atlas-large configuration with temporal extensions. It adapts T5-1.1 from the Fusion-in-Decoder architecture and leverages a dual-encoder architecture based on Contriever and a sequence-to-sequence model. The researchers replicated the pre-training for the generator and retriever from Atlas and experimented with different hyper-parameters to optimize TempRALM.
Measured against the original Atlas model, the San Jose State University team found that TempRALM outperformed the baseline model by up to 74%, while requiring less computational resources. TempRALM does not require retraining, recalculation, or any costly modifications to the document index.
The researchers plan to continue this work by exploring the relationship between LLMs and the retriever, and trialing various learning methods to fine-tune the parameters of the temporal relevance function. They suggest potential applications for their temporal retrieval method in recommender systems, fact-checking, and dialogue agents, among others.
