AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Ori Yoran , Samuel Joseph Amouyal , Chaitanya Malaviya , Ben Bogin , Ofir Press , Jonathan Berant
- 🏛 Institutions
- Tel Aviv University , University of Pennsylvania , Allen Institute for AI , University of Washington , Princeton
- 📅 Date
- October 21, 2024
- 📑 Publisher
- EMNLP 2024 (Poster)
- 💻 Env
- Web
- 🔑 Keywords
TLDR
Introduces AssistantBench, a benchmark of 214 realistic and time-consuming web tasks that require sustained planning, retrieval, and synthesis rather than short web interactions. The paper also proposes the SPA agent and shows that even strong models still struggle on these open-web tasks.
Related papers (24)
- ClawBench: Can AI Agents Complete Everyday Online Tasks?April 9, 2026 · arXiv
- The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying KnowledgeSeptember 24, 2026 · Findings of EMNLP 2026
- An Illusion of Progress? Assessing the Current State of Web AgentsApril 2, 2025 · COLM 2025
- CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use AgentsOctober 20, 2026 · arXiv
- PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience RetrievalSeptember 29, 2026 · arXiv
- LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual PuzzlesSeptember 28, 2026 · arXiv
- Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional FieldsJune 9, 2026 · arXiv
- ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental SynthesisMay 24, 2026 · arXiv
- CocoaBench: Evaluating Unified Digital Agents in the WildApril 13, 2026 · arXiv
- HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration TasksApril 10, 2026 · arXiv
- Gym-Anything: Turn any Software into an Agent EnvironmentApril 7, 2026 · arXiv
- AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI AgentsMarch 19, 2026 · arXiv
- OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive TasksJanuary 28, 2026 · arXiv
- MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World EnvironmentJanuary 28, 2026 · Findings of ACL 2026
- LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI AgentJanuary 26, 2026 · ICLR 2026 (Poster)
- MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented EnvironmentsDecember 22, 2025 · arXiv
- AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access BarriersOctober 29, 2026 · arXiv
- SkillLifeBench: Benchmarking Security Across the Lifecycle of Agent SkillsOctober 26, 2026 · arXiv
- AgentProofBench: Benchmarking Reusable Verifiers for Web AgentsOctober 25, 2026 · arXiv
- SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web AgentsOctober 24, 2026 · arXiv
- LoWAgent: A Low-Resource Multilingual Benchmark for Web AgentsOctober 19, 2026 · arXiv
- BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser ViewsOctober 12, 2026 · arXiv
- When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded TasksOctober 9, 2026 · arXiv
- A Baseline Assessment for the Agentic Web Technology StackOctober 9, 2026 · arXiv