AgentBench: Evaluating LLMs as Agents

Introduces AgentBench, a broad benchmark suite for evaluating LLMs as agents across eight environments, including but not limited to OS interaction. It belongs in this repo mainly as a general agent-evaluation reference rather than a pure GUI benchmark.

• GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
  April 6, 2026 · arXiv
• MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
  February 3, 2026 · arXiv
• An Illusion of Progress? Assessing the Current State of Web Agents
  April 2, 2025 · COLM 2025
• VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets
  September 11, 2026 · arXiv
• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
  September 9, 2026 · arXiv
• FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
  September 7, 2026 · arXiv
• APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  September 7, 2026 · arXiv
• ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults
  September 4, 2026 · arXiv
• GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
  August 30, 2026 · arXiv
• Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments
  August 25, 2026 · arXiv
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
  August 25, 2026 · arXiv
• Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  August 21, 2026 · arXiv
• ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
  August 18, 2026 · COLM 2026
• StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection
  August 6, 2026 · arXiv
• CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
  July 31, 2026 · arXiv
• OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
  July 30, 2026 · arXiv
• OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
  July 29, 2026 · arXiv
• EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
  July 19, 2026 · arXiv
• WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
  July 7, 2026 · arXiv
• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
  June 9, 2026 · arXiv
• Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms
  June 3, 2026 · arXiv
• Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
  May 26, 2026 · ICML 2026 (Poster)
• AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
  May 26, 2026 · arXiv
• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
  May 25, 2026 · arXiv