Amazon Bedrock, a full-service provider of high-functioning foundation models (FMs) utilizing artificial intelligence (AI) from leading companies, is improving data retrieval with a new feature: metadata filtering. Amazon Bedrock’s Knowledge Bases helps with Retrieval Augmented Generation (RAG), a method for fetching company data and enhancing prompts to provide more accurate results.
The new update announced on March 27, 2024, allows for the utilization of metadata fields during retrieval, a much-needed modification to mitigate accuracy issues when dealing with disparate data sets. Through metadata filtering, companies can more efficiently process large quantities of data and produce more accurate results from their FMs. Also, metadata fields on the Knowledge Bases need to be set up during the knowledge base ingestion process.
The upgraded metadata filtering workflow involves preparing data for metadata filtering, creating and ingesting data into the knowledge base, and then retrieving data from the knowledge base using metadata filtering.
Amazon Bedrock supports various underlying vector store providers, including Amazon OpenSearch Serverless, Amazon Aurora, Pinecone, Redis Enterprise, and MongoDB Atlas. For a simple demonstration, the framework was applied to tabular data in the public dataset Food.com – Recipes and Reviews and a detailed process was explained.
Notably, once the knowledge base has been created with the new metadata feature, data retrieval becomes much more accurate and focused. To further enable Amazon Bedrock’s capabilities, users have to specify metadata filtering conditions, such as “cholesterol content less than 10 and total preparation time of fewer than 30 minutes,” to get specific results.
The new feature also enables the use of the Retrieve API with these metadata filters. Plus, there’s now a retrieve_and_generate API that uses the same metadata filtering to pull relevant output.
The sample demonstration shared in the article revealed a significantly more accurate retrieval of records following the metadata filtering conditions specified. It helps users get more specific and useful results that align with their queries.
Although this functionality is useful, users are cautioned to delete any unused resources, as costs may be incurred for storing documents in the OpenSearch Serverless index.
In conclusion, the metadata filtering feature in Amazon Bedrock enhances the retrieval of information from large tabular data sets, making the results more accurate and relevant to the specified query. This advancement will open up new possibilities for leveraging Knowledge Bases in diverse fields.
