Web agents frequently face limitations due to their reliance on single input modalities and tests in controlled environments like web simulators or static snapshots. The results do not adequately represent the complexity and dynamism of real-world web interactions, restricting their real-life application and effectiveness in scenarios that require dynamic web content.
Prior studies in web agents have emphasized autonomous navigation and interaction with web environments. Important advancements include WebGPT and WebAgent, using GPT-3 and T5 models for text-based web browsing and HTML snippets extraction. There’s an increasing interest in multi-modal web agents, like WebGUM that combines T5 with Vision Transformers and PIX2ACT that uses web screenshots. Such efforts deviate from the previous single-modality or simplified web environment approaches, aiming for more realistic and dynamic web interactions. Concurrently, Large Multimodal Models (LMMs) (e.g., GPT-4V) are demonstrating robust multi-modal comprehension, which paves the way for more advanced web agents.
A collective work by researchers from Zhejiang University, Tencent AI Lab, and Westlake University introduces WebVoyager, an LMM-powered web agent that completes user instructions by interacting with real-world websites. A new evaluation method, leveraging GPT-4V’s robust multi-modal comprehension that includes a benchmark of tasks from 15 frequently visited websites, is proposed. WebVoyager’s interaction with the Apple website is demonstrated, showing an optimal path with zero redundancy.
WebVoyager reached a 55.7% task success rate – higher than GPT-4 and its text-only variant. The agent showed an 85.3% agreement rate with automatic evaluation utilizing GPT-4V, closely attuned to human judgment. Despite performing well on most website tasks, WebVoyager struggled with text-heavy websites like Cambridge Dictionary and Wolfram Alpha. However, the agent’s consistency improved markedly with additional information, reaching a Kappa score of 0.7, similar to human agreement level and revealing GPT-4V’s potential for efficient and large-scale web agent evaluations.
Looking ahead, possibilities for WebVoyager include refining integral methods for visual and textual information and exploring open-sourced LMMs for developing multi-modal web agents. The researchers have acknowledged that their work still needs development, as highlighted in the comprehensive Error Analysis provided in their paper.
The complete research paper is available to review. Kudos to the research team behind this innovation. For regular updates and future developments, you can follow us on Twitter or join our ML SubReddit, Facebook Community, Discord Channel, and LinkedIn Group. Our newsletter also provides valuable insights for those interested in our work. Remember to join our Telegram Channel. This post was first published on MarkTechPost.
