Structured Interaction, Visual Localization, and Robust Execution for Complex Web Tasks: A Technical Report on the WebRetriever Challenge

🏛 Institutions: Beijing Key Laboratory of Multimodal Super-Intelligent Security , People AI
📅 Date: September 28, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: WebRetriever Challenge technical report visual localization context management

TLDR

Technical report of the winning system in the WebRetriever Challenge. It uses semantic webpage information for routine browser operations and invokes vision only when structured representations fall short, with grid-assisted visual localization, hierarchical context management, and fault-aware execution across eight concurrent browser workers; it achieved a 59% pass rate on the official Protocol 3.

Open paper Report issue