The improvement of logical reasoning capabilities in Large Language Models (LLMs) is a critical challenge for the progression of Artificial General Intelligence (AGI). Despite the impressive performance of current LLMs in various natural language tasks, their limited logical reasoning ability hinders their use in situations requiring deep understanding and structured problem-solving.
The need to overcome this barrier is pressing, given the rising demand for AI systems capable of handling complex reasoning tasks in various sectors, including natural language processing, automated reasoning, robotics, and scientific research. For example, despite their innovations, methods such as Logic-LM and CoT have shortcomings in managing intricate reasoning tasks. Logic-LM relies on external solvers for translation, which might lead to data loss. In comparison, CoT finds it challenging to strike a balance between precision and recall, affecting its overall logical reasoning performance.
In response to this, researchers from the National University of Singapore, the University of California, and the University of Auckland have introduced the Symbolic Chain-of-Thought (SymbCoT) framework. This technique blends symbolic expressions with CoT prompting to improve logical reasoning in LLMs. By including symbolic representation and rules, SymbCoT can enhance reasoning significantly, offering a more versatile and efficient solution for complex reasoning tasks than current methods like CoT and Logic-LM.
SymbCoT uses symbolic structures and rules to guide the reasoning process, improving the model’s capacity to undertake intricate logical tasks. The framework adopts a ‘plan-then-solve’ strategy, breaking down questions into smaller parts for efficient reasoning. It also outlines the computational resources required for its implementation, indicating its scalability and practicality.
SymbCoT is shown to perform significantly better than the Naive, CoT, and Logic-LM baselines, scoring higher by 21.56%, 6.11%, and 3.53% on GPT-3.5, and 22.08%, 9.31%, and 7.88% on GPT-4, respectively. However, it struggled to beat Logic-LM with the FOLIO dataset on GPT-3.5, suggesting challenges with non-linear reasoning in LLMs. Nonetheless, consistently outperforms all the baselines across both datasets with GPT-4, particularly surpassing Logic-LM by an average of 7.88%.
In sum, the SymbCoT framework offers a substantial advancement in AI research by enhancing logical reasoning capabilities in LLMs. The findings of this paper suggest broad implications for AI applications. Future research could focus on investigating additional symbolic languages and optimizing the framework for broader adoption in AI systems. This research presents a solution to a critical challenge in logical reasoning, potentially paving the way for more advanced AI systems with improved reasoning capabilities.
