Recent progress in large language models (LLMs) has advanced our ability to interpret and implement instructions. However, LLMs still struggle with recall and composition of world knowledge which results in inaccurate responses. A suggested approach to improve reasoning is the integration of auxiliary tools such as search engines or calculators during inference. Existing tool-augmented LLMs have issues with efficiently using these tools for multi-step reasoning and minimizing inference waiting times.
In a collaborative research by EPFL and Meta, a novel method known as Chain-of-Abstraction (CoA) reasoning is introduced for resolving these challenges. LLMs are trained to create reason chains with abstract placeholders, which are replaced by specific knowledge from external tools for the final answer generation.
The uniqueness of CoA reasoning lies in the separation of general reasoning from domain-specific knowledge leading to parallel processing, which speeds up the overall inference process. Unlike earlier methods that interspersed LLM decoding and API calls, CoA advocates connecting multiple tool calls and adopting more feasible reasoning strategies.
For implementing CoA reasoning, existing open-source question-answering datasets are utilized to train LLMs. The CoA method is tested on mathematical reasoning and Wikipedia question answering (Wiki QA). In both areas, significant improvements in performance were recorded, with an average accuracy increase of about 7.5% in mathematical reasoning and 4.5% in Wiki QA respectively.
The CoA method has paved the way for more robust multi-step reasoning in LLMs. It not only enhances accuracy, but also increases reasoning speed, which makes it an excellent tool for a wide range of scenarios. This research highlights the potential of CoA for broader applications in improving LLM performance across various domains.
The post was originally published on MarkTechPost.
