REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
Divyansh Garg , Shaun VanWeelden , Diego Caples , Andis Draguns , Nikil Ravi , Pranav Putta , Naman Garg , Tomas Abraham , Michael Lara , Federico Lopez , James Liu , Atharva Gundawar , Prannay Hebbar , Youngchul Joo , Jindong Gu , Charles London , Christian Schroeder de Witt , Sumeet Motwani
- 🏛 Institutions
- The AGI Company , Stanford , Oxford , Mercor , Contramont Research , Plato , Independent
- 📅 Date
- April 15, 2025
- 📑 Publisher
- arXiv
- 💻 Env
- Web
- 🔑 Keywords
TLDR
REAL benchmarks autonomous web agents on deterministic replicas of 11 real websites so evaluation stays realistic while remaining safe and reproducible. It pairs 112 practical multi-turn tasks with an evaluation harness that mixes programmatic state checks and rubric-guided LLM judgments, and reports frontier agents reaching only about 41% success.
Related papers (24)
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal ModelsJanuary 25, 2024 · ACL 2024
- APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI AgentsSeptember 7, 2026 · arXiv
- SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent EvaluationOctober 19, 2024 · ICLR 2025 (Spotlight)
- AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access BarriersOctober 29, 2026 · arXiv
- SkillLifeBench: Benchmarking Security Across the Lifecycle of Agent SkillsOctober 26, 2026 · arXiv
- AgentProofBench: Benchmarking Reusable Verifiers for Web AgentsOctober 25, 2026 · arXiv
- SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web AgentsOctober 24, 2026 · arXiv
- LoWAgent: A Low-Resource Multilingual Benchmark for Web AgentsOctober 19, 2026 · arXiv
- BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser ViewsOctober 12, 2026 · arXiv
- When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded TasksOctober 9, 2026 · arXiv
- A Baseline Assessment for the Agentic Web Technology StackOctober 9, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- Securing Computer-Use Agents Against Branch Steering AttacksOctober 2, 2026 · NeurIPS 2026 Workshop on Agents in the Wild
- AdaptArena: Evaluating Test-Time Personalization of Web AgentsSeptember 29, 2026 · arXiv
- WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web AgentsSeptember 28, 2026 · arXiv
- DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard AnalysisSeptember 26, 2026 · arXiv
- AgentTell: Behavioural Side-Channel Leakage in Browser-Use AgentsSeptember 26, 2026 · arXiv
- The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying KnowledgeSeptember 24, 2026 · Findings of EMNLP 2026
- Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI AgentsSeptember 23, 2026 · ACM MM 2026
- Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web AgentsSeptember 23, 2026 · arXiv
- CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned EnvironmentsSeptember 23, 2026 · arXiv
- Constructing Challenging Browser-Use Tasks by Controlled Environment InterventionsSeptember 20, 2026 · arXiv
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv