Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories
Chengguang Gan , Zimeng He , Yoshihiro Tsujii , Ken-ichiro Kobayashi , Hiroki Itoh , Kotaro Funakoshi
- 🏛 Institutions
- Techtouch , Institute of Science Tokyo
- 📅 Date
- October 1, 2026
- 📑 Publisher
- NeurIPS 2026 Workshop on Who Verifies the Agents? Toward Reliable Agent Development
- 💻 Env
- Web
- 🔑 Keywords
TLDR
Audits all 165 WebArena-Lite tasks under six evaluation conditions with human review of outcomes and trajectories. Human review recovers successes the automatic evaluator missed, a failure analysis of 102 trajectories identifies recurring error patterns, and a memory-and-analysis mechanism and guide text improve success.
Related papers (24)
- WebATLAS: An LLM Agent with Experience-Driven Memory and Action SimulationOctober 26, 2025 · NeurIPS 2025 Workshop on Language Agents and World Models
- MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic EnvironmentsFebruary 3, 2026 · arXiv
- Interaction-as-a-JudgeOctober 13, 2026 · arXiv
- A Baseline Assessment for the Agentic Web Technology StackOctober 9, 2026 · arXiv
- Harness Evolution Hits a Ceiling: When Weight Training Should BeginOctober 8, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live WebsitesOctober 2, 2026 · arXiv
- Action Conditioned Bisimulation For GUI Agent MemorySeptember 30, 2026 · arXiv
- WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web AgentsSeptember 28, 2026 · arXiv
- GUITAR: Structured Failure Diagnosis of GUI Agents via State TransitionsSeptember 28, 2026 · arXiv
- Probe to Act: Elevating Browser-Use Agent via Active Visual ProbingSeptember 27, 2026 · Findings of EMNLP 2026
- From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMsSeptember 26, 2026 · arXiv
- Constructing Challenging Browser-Use Tasks by Controlled Environment InterventionsSeptember 20, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv
- Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search SystemsApril 9, 2026 · arXiv
- Why Do LLM-based Web Agents Fail? A Hierarchical Planning PerspectiveMarch 15, 2026 · arXiv
- Hybrid Self-evolving Structured Memory for Computer-Use AgentsMarch 11, 2026 · Findings of ACL 2026
- Enhancing Web Agents with a Hierarchical Memory TreeMarch 7, 2026 · arXiv
- Mobile-Agent-v3.5: Multi-platform Fundamental GUI AgentsFebruary 15, 2026 · arXiv
- Agentic Test-Time Scaling for WebAgentsFebruary 12, 2026 · arXiv
- WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph RepresentationOctober 22, 2025 · NeurIPS 2025 Workshop on Multi-Turn Interactions in Large Language Models
- SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web AgentsOctober 11, 2025 · arXiv
- JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents AdaptationOctober 5, 2025 · arXiv
- WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement LearningMay 22, 2025 · EMNLP 2025 (Poster)