Decision-making is crucial for organizations, often requiring data analysis and selection processes to determine the best alternative to meet specific objectives. For instance, pharmaceutical distribution networks often have to confront daunting decisions such as choosing the appropriate plants to run, deciding on the number of employees to employ, and optimizing production costs while ensuring prompt delivery. The decision-making task typically involves planning the necessary analysis, retrieving applicable data, and making decisions based on that data. Although decision support systems have assisted with the latter two steps, the critical first step of planning the necessary analysis remains a predominantly human-led process. This process still has human-driven attributes despite the advancements in automation technology, and achieving complete end-to-end decision-making without human interference presents considerable challenges.
Given these challenges, researchers at the School of Computing, KAIST have proposed a new task referred to as “Decision QA” that works toward equipping large language models (LLMs) with the ability to make optimal decisions by analyzing structured data and business rules. The Decision QA is formulated as a question-and-answer-style (QA) task with the objectives of taking a database, business rules, and a decision-making question as inputs to generate the best possible decision. The benchmark for this task, called DQA, includes two scenarios – the Locating scenario, which involves questions about the best placement of resources, and the Building scenario, which tackles questions related to resource allocation. The benchmark uses data taken from strategy video games simulating real-world business situations.
To handle the Decision QA task more effectively, the researchers proposed a method called “PlanRAG”. The PlanRAG method includes Planning, where the LLM takes the decision-making question, database schema, and business rules as inputs, then generates an initial plan for the necessary data analyses for decision-making. This is followed by a Retrieving and Answering step that uses the data analysis prepared in the initial plans. Finally, if the initial plan is insufficient, it proceeds to Re-planning, wherein the LLM creates another plan for further analysis.
The PlanRAG method proves significantly more efficient in boosting the decision-making performance of LMs compared to the present advanced iterative RAG technique. PlanRAG not only handles simple and complex decision-making tasks effectively, but also excels by 15.8% in the Locating scenario and 7.4% in the Building scenario compared to its competition. The PlanRAG model’s strategic approach to planning, systematic data retrieval, and significantly lower rates of missed data analysis gives it an advantage. Especially notable is its exemplary performance in complex settings that require multi-hop reasoning on graph databases.
This study explores the application of LLMs for complex decision-making tasks and proposes the Decision QA task and the PlanRAG technique. Consequently, it provides an exciting prospect for decision-making applications involving LLMs and structured data.
