Artificial Intelligence (AI) aims to automate computer operations through autonomous agents that can reason, plan, and act. The challenge lies in developing agents that can manage computers, process a range of inputs, understand complex natural language commands, and perform tasks to meet set goals. Currently, the focus has predominantly been placed on text-based agents.
To address these challenges, Carnegie Mellon University researchers introduced VisualWebArena, a benchmark created to assess the performance of multimodal web agents on challenging, realistic, and visually complex tasks. The benchmark tests the agents’ ability to interpret image-text inputs, understand natural language instructions, and perform tasks on websites to achieve user-set goals.
The benchmark includes 910 realistic tasks in three different online contexts – Reddit, Shopping, and Classifieds. The Shopping and Reddit contexts extend from WebArena, while Classifieds presents as a new addition. Unlike WebArena, tasks in VisualWebArena are visually anchored, necessitating a thorough understanding of the content.
A detailed evaluation was carried out on the most advanced Large Language Model (LLM)-based autonomous agents, including many multimodal models. The study identifies the capabilities gap of the most advanced multimodal language agents, offering valuable insights for future developments.
The research compared performance of LLM and Vision-Language Models (VLMs). Results showed that VLMs perform much better than text-only LLMs, attaining a success rate of 16.4%, markedly below the human performance of 88.7%.
The study further disclosed a significant discrepancy between open-sourced and API-based VLM agents, emphasizing the need for comprehensive assessment metrics. A new VLM agent, which employs the Set-of-Marks prompting strategy, demonstrated significant performance benefits on visually complex web pages.
In summary, VisualWebArena provides a framework for evaluating multimodal autonomous agents and knowledge to drive the development of improved, autonomous agents for tackling complex web tasks. The benchmark uniquely enables a comprehensive understanding of the potentials and limitations of autonomous agents, offering a valuable platform for future AI developments. The original study can be found on Github. Credit for this research goes to the associated team of researchers.
