Large Language Models (LLMs), AI models capable of performing general tasks by prompting, have gained prominence in 2023. Even as these models continue to improve, there has been a significant shift towards compound AI systems, which integrate multiple components. This shift has led to notable AI results and made it possible to obtain better AI outputs with clever engineering rather than by merely increasing the model’s size. Compound AI systems assemble multiple interacting AI models, retrievers, or external tools to solve AI tasks, contrasting with conventional AI models that predict the next sequence in text.
This article outlines the reasons for the growing interest in and application of compound AI systems. Notably, compound systems are more useful for various tasks as they provide enhanced control and trust. Improving such systems is often more efficient and faster than training single models. Compound systems can also dynamically incorporate timely data, improve AI control, and adjust performance according to specific goals.
The development of compound AI systems is not without challenges. The systems consist of components that are non-differentiable such as search engines or code interpreters, requiring new optimization methods. Developers also need to explore a vast design space to craft a suitable system. Lastly, managing machine learning operations becomes more complex with compound AI systems.
To tackle these challenges, new frameworks and strategies are emerging for designing compound AI systems. For automatically optimizing quality, frameworks like DSPy tune the pipeline to maximize the end-to-end performance of an application. For optimizing cost, frameworks like FrugalGPT learns a routing strategy that improves the quality of AI services while reducing cost. To aid operation, tools like LangSmith and Phoenix Traces provide fine-granularity tracking, visualization, and evaluation of outputs.
The authors anticipate that despite the advancement of AI models, compound AI systems will remain central to maximizing the quality and reliability of AI applications. Therefore, they urge developers to explore and leverage emerging frameworks to optimize these compound systems.
