Carnegie Mellon University researchers have introduced VisualWebArena, a benchmark meant to assess the performance of AI agents on web-based tasks by offering realistic and visually stimulating challenges. Current AI benchmarks largely focus on text-based capabilities of agents, but VisualWebArena provides a platform for evaluating agents on their understanding of image-text inputs, comprehension of natural language instructions, and their capabilities of performing tasks on websites.
Through an evaluation carried out on Large Language Model (LLM)–based autonomous agents, the team found that text-only agents had certain limitations. It also identified gaps in the most advanced multimodal language agents.
VisualWebArena includes 910 realistic activities across three different online environments: Reddit, Shopping, and Classifieds. Unlike previous benchmarks like WebArena, all tasks in VisualWebArena are visually anchored, with a significant portion requiring understanding interleaving.
In their comparison of state-of-the-art Large Language Models and Vision-Language Models (VLMs), the team found that the latter outperformed the former. However, the highest-achieving VLM agents only managed a success rate of 16.4%, far below human performance (88.7%). It was also discovered an important discrepancy between open-sourced and API-based VLM agents. The team proposed a unique VLM agent, inspired by the Set-of-Marks prompting strategy, which improved performance on complex web pages.
In summary, VisualWebArena provides a benchmark for evaluating multimodal autonomous language agents. The findings from the evaluations will be useful in developing more robust and capable autonomous agents for online tasks.
