Recent advances in language models (LMs) and tools have paved the way for semi-autonomous agents such as WebGPT, AutoGPT, and ChatGPT plugins that operate in real-world settings. However, transitioning from text interactions to real-world actions poses unique risks, including potential financial losses, property damage, or even life-threatening situations. It is of utmost importance to identify these risks before the deployment of LM agents.
These risks are challenging due to their open-ended nature and the considerable engineering work required for testing. Traditional testing procedures are labor-intensive, inhibiting scalability and the identification of long-tail risks. In response to these challenges, a strategy from simulator-based testing in high-risk areas is adopted: introducing ToolEmu, an LM-based tool emulation framework. The framework is designed to investigate LM agents across various tools, identify failures, and help develop safer agents using an automatic evaluator.
ToolEmu uses an LM, like GPT-4, to mimic tools and their execution sandboxes. Unlike conventional simulated environments, ToolEmu allows the quick prototyping of LM agents across scenarios, even for high-risk tools without existing APIs or sandbox implementations. To reinforce risk analysis, an adversarial emulator is introduced to spot potential agent failure modes. In testing ToolEmu, over 80% of 200 tool execution trajectories were deemed realistic by human evaluators, with 68.8% of failures acknowledged as truly risky.
Supporting scalable risk assessments, the safety evaluator within the framework measures potential failures and related risk severities and identifies 73.1% of failures. A trade-off between safety and helpfulness is quantified using an automatic helpfulness evaluator.
The emulators and evaluators facilitate the creation of a benchmark for quantitative evaluations of LM agents across various tools and scenarios. The benchmark includes 144 test cases across nine risk types and 36 tools, focusing on a threat model containing unclear user instructions. Evaluations demonstrate that API-based LMs like GPT-4 and Claude-2 achieve top scores in both safety and helpfulness. However, even the safest LM agents exhibit failures in 23.9% of test cases, stressing the need for ongoing efforts to increase safety.
All credit for the research goes to its authors and supporters. To keep updated on this and other projects, follow us on our social media platforms and join our Telegram channel. If you enjoyed our work, you’ll love our newsletter.
