Qdrant, a pioneer in vector search technology, has unveiled BM42, a powerful new algorithm, aimed at transforming hybrid search. BM25, the algorithm relied upon by search engines like Google and Yahoo, has dominated for over 40 years. Yet, the rise of vector search and the launch of Retrieval-Augmented Generation (RAG) technologies have revealed the need for a more sophisticated solution. BM42 aims to fill this void by merging the potency of BM25 with contemporary transformer models, a significant enhancement for search applications.
BM25’s ongoing relevance is credited to its straightforward but effective algorithm, which estimates the significance of documents based on term frequency and inverse document frequency. This method works well in traditional web search scenarios where document length and query structures are predictable. Nevertheless, the text retrieval landscape has evolved considerably with the advent of RAG systems. These systems necessitate the management of shorter, more diverse documents and queries, an area where BM25’s reliance on document statistics is less efficient.
To overcome these challenges, BM42 integrates the fundamental principles of BM25 with the abilities of transformer models. BM42’s major innovation is utilizing attention matrices from transformers to determine a term’s significance within a document. Transformers generate several outputs, one of them being attention matrices which spotlight each token’s importance in the input sequence. By leveraging the attention row that corresponds to the peculiar [CLS] token, BM42 can correctly assess the importance of each token in a document, even for shorter texts commonplace in RAG applications.
BM42 comes with numerous advantages over BM25 and SPLADE, another recent alternative that employs transformers to generate sparse embeddings. While SPLADE has demonstrated superior performance in academic benchmarks, it calls for improvement in its performance metrics, most notably the need for intensive computational resources and difficulties with tokenization and domain dependency. In contrast, BM42 maintains the simplicity of BM25 and addresses the shortcomings of SPLADE.
A standout feature of BM42 is its efficiency. It can perform document and query deductions rapidly, making it useful for real-time applications. Additionally, it has a small memory footprint, which makes it capable of handling large datasets without excessive resource demands. With the availability of a suitable transformer model, BM42 supports multiple languages and domains, thus, highly versatile.
Implementing BM42 in Qdrant’s vector search engine is straightforward. The process involves setting up a collection for hybrid search with BM42 and using dense embeddings from models like jina.ai. By maintaining a balanced approach where sparse and dense embeddings reinforce each other, retrieval accuracy is improved. Qdrant’s benchmarks indicate that BM42 surpasses BM25 in situations involving short texts, a frequent requirement in modern search applications.
Qdrant’s launch of BM42 not only provides a new and superior algorithm, but also promotes community engagement and innovation. The company encourages developers and researchers to experiment with BM42 and contribute to its ongoing development. Qdrant is keen on empowering its community to push beyond the boundaries of what’s possible in search technology.
In conclusion, Qdrant’s release of BM42 is a significant event in the evolution of search algorithms. By integrating the strength of BM25 with the ingenuity of transformers, BM42 establishes a new benchmark for hybrid search. It overcomes the weaknesses of previous models and current alternatives by offering a versatile, efficient, and highly precise solution for today’s search requirements.
