AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
Xing Han Lù , Dheeraj Vattikonda , Sina Hajimiri , Fatemeh Pesaran Zadeh , Parishad BehnamGhader , Ghazwa Darwiche , Amirhossein Kazemnejad , Christopher Pal , Alexandre Drouin , Siva Reddy
- 🏛 Institutions
- ServiceNow Research , McGill University , Mila , Polytechnique Montréal , SNU , Université Laval , ÉTS Montréal
- 📅 Date
- October 8, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop Web
- 🔑 Keywords
TLDR
Presents AgentHorizon, a benchmark and diagnostic suite evaluating automated agentic judges on multi-application, long-horizon computer-use trajectories from OSWorld and WorkArena. Shows that current VLM judges suffer significant false-positive rates on complex multi-window workflows due to temporal aggregation errors and misread intermediate states.
Related papers (24)
- CUA-Lite: An Open Platform for Computer Use AgentsOctober 19, 2026 · arXiv
- PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse PersonasOctober 16, 2026 · arXiv
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon AgentsOctober 13, 2026 · arXiv
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsApril 11, 2024 · NeurIPS 2024 Datasets and Benchmarks Track
- Agent World Model in the Wild: Computer-Use Agent Environment SynthesisOctober 19, 2026 · arXiv
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer UseOctober 15, 2026 · arXiv
- OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using AgentsOctober 15, 2026 · arXiv
- LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use AgentsOctober 15, 2026 · arXiv
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer EnvironmentsOctober 14, 2026 · arXiv
- A Baseline Assessment for the Agentic Web Technology StackOctober 9, 2026 · arXiv
- Control OSWorld: An AI Control Environment for GUI Computer Use AgentsOctober 2, 2026 · arXiv
- Talk2Agent: Benchmarking Voice Interfaces for Text AgentsSeptember 30, 2026 · arXiv
- OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific SoftwareSeptember 30, 2026 · arXiv
- WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web AgentsSeptember 28, 2026 · arXiv
- OSWorld-Pro: Process-based Evaluation for Computer Use AgentsSeptember 21, 2026 · arXiv
- Odysseys: Benchmarking Web Agents on Realistic Long Horizon TasksApril 27, 2026 · arXiv
- An Illusion of Progress? Assessing the Current State of Web AgentsApril 2, 2025 · COLM 2025
- Learning How to Remember: Memory Modality Selection for GUI AgentsOctober 13, 2026 · arXiv
- Interaction-as-a-JudgeOctober 13, 2026 · arXiv
- Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill InductionOctober 9, 2026 · arXiv
- Latent World Action Model for GUI AgentsOctober 9, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- CUAWright: A Minimal Unified Interface for Digital AgentsOctober 2, 2026 · arXiv
- cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use AgentsSeptember 30, 2026 · arXiv