Large language models (LLMs), such as those used in AI, can creatively solve complex tasks in ever-changing environments without the need for task-specific training. However, achieving broad, high-level goals with these models remain a challenge due to the objectives’ ambiguous nature and delayed rewards. Frequently retraining models to fit new goals and tasks is also not always feasible.
Existing solutions offer aid in two forms: prior task decomposition and post-hoc experience summarization. However, these methods face limitations, including a lack of empirical grounding (prior task decomposition) and difficulty in effectively prioritizing strategies (post-hoc experience summarization).
Several methods have been explored in prior studies to alleviate these issues. Reflexion enables agents to reflect on their failures and come up with new plans while Voyager provides a library of code-based skills learned from detailed feedback. Analyzing both failed and successful attempts to achieve goals aids in summarizing causal abstractions. However, feedback is often general and unsystematic. These limitations have led to the development of more adaptive and context-aware approaches to allow autonomous language agents to achieve high-level goals.
As part of a joint effort from Fudan University and the Allen Institute for AI, researchers proposed SELFGOAL. This self-adaptive framework allows language agents to employ both prior knowledge and environmental feedback to achieve high-level goals. It does this by constructing a tree of textual sub-goals, with appropriate ones chosen based on the current situation. SELFGOAL contains two main modules: a Search Module that selects the most suited goal nodes, and a Decomposition Module that breaks these goal nodes down into more detailed sub-goals. An action module then uses these selected sub-goals to guide the LLM in taking action. This method provides more straightforward guidance for high-level goals and can adapt to different environments.
SELFGOAL follows a top-down hierarchical decomposition of the high-level goal, with three central modules: Search, Decomposition, and Action. The Search Module finds the most suitable sub-goals for the circumstance. The Decomposition Module refines these sub-goals into more concrete ones, using a filtering function to prevent redundancy. The Act Module then uses these refined sub-goals to update the instruction prompt and guide the agent’s actions. This technique enables SELFGOAL to adapt to changing situations and provide accurate guidance.
In testing, SELFGOAL significantly outperforms other baseline frameworks, demonstrating greater improvement as the LLMs become larger. Unlike current decomposition methods which give general or unsuitable guidance, or post-hoc experience summarization methods that can be overly detailed, SELFGOAL dynamically adjusts itself. In instances such as the Public Good Game, SELFGOAL adjusts its sub-goals based on observed player behaviors, letting agents adapt their strategies effectively. Its structural architecture also allows it to perform well with smaller LLMs and outdo its competition, such as in auction competitions.
The researchers who developed SELFGOAL believe it significantly improves an LLM agent’s ability to achieve high-level goals across various tasks and environments. It does this by continually refining and generating a GOALTREE, a hierarchical collection of contextual sub-goals, based on environmental interactions, improving agent performance in both competitive and collaborative situations. More work still needs to be done to fully realize its potential, but even for smaller models, SELFGOAL is a considerable advance in achieving high-level goals without regular retraining.
