In the field of large language models (LLMs), multi-agent debates (MAD) pose a significant challenge due to their high computational costs. They involve multiple agents communicating with one another, all referencing each other’s solutions. Despite attempts to improve LLM performance through Chain-of-Thought (CoT) prompting and self-consistency, these methods are still limited by the increased complexity and high computational resource demands associated with expanded input context deriving from multiple debates.
Considering this, Google DeepMind’s researchers have proposed a novel solution using sparse communication topology in multi-agent debates. The main goal is to maintain or even improve MAD performance while considerably reducing computational costs. This is achieved by employing a neighbor-connected communication strategy where agents interact with a restricted group of peers rather than everyone. This significantly reduces the input context size, making the debate process more efficient and scalable.
Static graphs are used in this method to illustrate the communication topologies between agents – their relationships quantified by a sparsity ratio. The focus is mainly on configurations with six agents and varying degrees of sparsity. The agents, instantiated with models such as GPT-3.5 and Mistral 7B, engage in multiple rounds of debate, continuously refining their answers based on their peers’ responses. Various datasets are used, including the MATH and GSM8K for reasoning tasks, and Anthropic-HH for alignment labeling tasks. Performance is measured through accuracy and cost savings, with robust outcomes ensured by employing variance reduction techniques.
Results have shown improvements in both MAD performance and computational efficiency with the use of a sparse communication topology. For example, a neighbor-connected topology within the MATH dataset improved accuracy by 2% compared to fully connected MAD, and reduced the average input token cost by over 40%. In alignment labeling tasks using the Anthropic-HH dataset, sparse MAD configurations demonstrated improvements and halved computational costs. These findings demonstrate that sparse communication topologies can achieve performance comparable or even superior to fully connected topologies, but at significantly lower computational costs.
Overall, this research introduces a valuable breakthrough in AI by incorporating a sparse communication topology in multi-agent debates. This method efficiently addresses the computational inefficiencies of existing methods and provides a scalable and more resource-efficient solution. The potential impact of this advancement is demonstrated in the experimental results – showing an enhancement in performance while reducing costs. This showcases the potential of multi-agent systems in practical applications.
