At the Data + AI Summit 2024, Databricks unveiled the public preview of the Mosaic AI Agent Framework and Agent Evaluation, aimed at helping developers build and deploy superior Agentic and Retrieval Augmented Generation (RAG) applications on the Databricks Data Intelligence Platform.
Building quality generative AI applications pose distinct challenges for developers, such as selecting the right metrics to assess application quality, effectively collecting user feedback, identifying the origin of performance issues, and effective problem-solving. The Mosaic AI Agent Framework and Agent Evaluation address these issues through various unique capabilities.
The Agent Evaluation tool integrates human feedback, allowing developers to bring in experts across their organization to review and provide feedback on the AI applications, even if these individuals are not Databricks users. This approach helps gather a wide range of perspectives, refining product quality.
Created in partnership with Mosaic Research, Agent Evaluation provides a set of metrics for gauging application quality, incorporating accuracy, hallucination, helpfulness, and harmfulness into its measures. A system logs responses and feedback to an evaluation table for analysis and problem solve quality issues. The integration of the Agent Framework with MLflow supports the shift from development to production.
As part of app lifecycle management, the Agent Framework offers a simplified SDK for managing applications throughout production, including permissions management and deployment through Mosaic AI Model Serving.
Databricks presented an example of the use of the Mosaic AI Agent Framework in the creation of a high-quality RAG application. This case shows a simple build of a RAG application that retrieves relevant chunks of data from a vector index and summarizes these for user queries, demonstrating the ease of building, critiquing, and improving generative AI applications using the Mosaic AI tools.
Several companies have benefited from the implementation of the Mosaic AI Agent Framework in their AI solutions. For example, Corning used the system to create an AI research assistant that data indexes, enhancing retrieval speed, response quality, and accuracy.
Pricing for the Mosaic AI tools is based on judge requests for Agent Evaluation, while rates for Mosaic AI Model Serving depend on Mosaic AI Model Serving rates. Various resources about how to use and implement these tools correctly are available, such as Accessible Agent Framework documentation, Generative AI Cookbook, and demo notebooks. In short, Databricks Mosaic AI encompasses the critical tools necessary for developers to build, evaluate, and deploy high-quality generative AI applications. By providing comprehensive solutions to common challenges, Databricks empowers developers to innovate and meet high-quality and performance standards.
