Together AI has announced an advancement in artificial intelligence with a new approach called the Mixture of Agents (MoA), also referred to as Together MoA. This model employs the combined strengths of multiple large language models (LLMs) to deliver increased performance and quality, setting a new standard for AI.
The MoA’s design incorporates layers, each containing multiple LLM agents. These agents leverage the outputs from the preceding layer as additional information to refine their responses. The ability to assimilate insights and capabilities from several models means the combined model is both robust and versatile. This approach has shown a high level of success, with a notable 65.1% score on the AlpacaEval 2.0 benchmark. This result surpasses the previous top score achieved by GPT-4o, which was 57.5%.
The development of MoA has been driven by the concept of collaborativeness among LLMs. The idea is that an LLM can produce improved responses when given outputs from other models, even lesser-performing ones. Utilizing this collaborative concept, MoA’s design divides models into ‘proposers’ and ‘aggregators.’ Proposers generate initial responses, while aggregators refine these into high-quality outputs. This iterative process continues until a final, polished response is generated.
Multiple benchmarks have rigorously tested the Together MoA framework, including MT-Bench, FLASK, and AlpacaEval 2.0. The results have been impressive, with Together MoA achieving leading positions on both the MT-Bench and AlpacaEval 2.0 leaderboards. Remarkably, on the AlpacaEval 2.0, Together MoA achieved a 7.6% absolute improvement over GPT-4o, scoring 65.1% by using only open-source models.
Beyond its technical achievements, cost-effectiveness is a key feature of Together MoA. Research suggests that Together MoA’s configurations offer the best balance between high-quality results and cost. This is evident in the Together MoA-Lite configuration, which matches the cost of GPT-4o but delivers superior quality despite having fewer layers.
The success of MoA is accredited to the collaborative efforts of several organizations in the open-source AI community. These include Microsoft, Alibaba Cloud, Meta AI, Mistral AI, and DataBricks. Their contributions have been crucial, and the benchmarks developed by LMSYS, Tatsu Labs, and KAIST AI have played a key role in measuring MoA’s performance.
Together AI intends to continue speeding up response times and further optimizing the MoA architecture by exploring different configurations, model choices, and prompts. The focus is on enhancing MoA’s skills in reasoning-focused tasks, reinforcing its role as a leader in AI innovation.
In conclusion, Together MoA signifies an important advance in harnessing the combined capabilities of open-source models. Its layered approach and collaborative principle underscore the potential for further enhancing AI systems making them more capable, robust, and in sync with human reasoning. The AI community eagerly looks forward to witnessing the continued evolution and application of this groundbreaking technology.
