Large Language Models (LLMs) have shown great potential in natural language processing tasks such as summarization and question answering, using zero-shot and few-shot prompting approaches. However, these prompts are insufficient for enabling LLMs to operate as agents navigating environments to carry out complex, multi-step tasks. One reason for this is the lack of adequate training data for fine-tuning these models. Gathering data for intricate, decision-making tasks is both time-consuming and costly. Moreover, automatically evaluating the sequence of actions taken by an agent remains a challenging task due to metrics limitations.
Self-improvement techniques for LLMs have been proposed, which include self-distillation where the teacher and student are the same models. Also, performance can be improved using multiple prompting methods. Self-improving agents represent a new way to tackle complex tasks, learning and improving through individual means. Although filtering and fine-tuning trajectories is a feature of one method, the emphasis is on supervised filtering and does not involve generating novel tasks or synthetic data.
Researchers from several institutions, including the University of Pennsylvania and ExtensityAI, have developed new techniques that allow LLM agents to tackle complex tasks through self-improvement. Key to this is the fine-tuning of the LLM and employing unsupervised learning methods, such as self-critique to filter training examples. Two auxiliary metrics were also introduced to analyze the capabilities gained or lost by the agent and to measure the quality of agent trajectories of different lengths.
When applied, these metrics captured small yet significant changes that added more value than the overall benchmark scores. Additionally, a series of experiments were conducted to fine-tune agent models using synthetic training data and evaluate the self-improvement of the agent model. These comparisons showed a considerable improvement in performance. The results of the experiments indicate that models can self-improve web agent tasks and enhance the overall benchmark performance. For instance, one experiment resulted in an agent solving 18 tasks correctly, culminating in a relative improvement of 31%.
All in all, the study showed that self-improving LLM agents can gain new capabilities and perform complex tasks more efficiently. However, the limitations of the fine-tuning methods used, which tend to reinforce not only correct actions and decisions but also the incorrect ones of the underlying model, require further development. This weakness can potentially be remedied through the application of human or supervised filtering.
