A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation
Yuxiang Chai , Shunye Tang , Han Xiao , Weifeng Lin , Hanhao Li , Jiayu Zhang , Liang Liu , Pengxiang Zhao , Guangyi Liu , Guozhi Wang , Shuai Ren , Rongduo Han , Haining Zhang , Siyuan Huang , Hongsheng Li
- 🏛 Institutions
- CUHK , vivo AI Lab , SJTU
- 📅 Date
- January 2, 2025
- 📑 Publisher
- arXiv
- 💻 Env
- Mobile
- 🔑 Keywords
TLDR
A3 is a mobile GUI benchmark built from 100 tasks over 20 dynamic online Android apps to evaluate agents beyond static or offline settings. Its essential-state procedural evaluation uses MLLMs as reward models to verify both intermediate progress and final completion on real online apps.
Related papers (24)
- AndroidFlux: Evaluating Failure Recovery and Recovery Action Reward Modeling in Mobile-Use Multimodal AgentsOctober 25, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv
- The Art of Building Verifiers for Computer Use AgentsApril 5, 2026 · arXiv
- CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using AgentOctober 21, 2025 · arXiv
- Web-Shepherd: Advancing PRMs for Reinforcing Web AgentsMay 21, 2025 · NeurIPS 2025 (Spotlight)
- iPrefBench: A User-Centric Benchmark for Personalized Mobile Use AgentsOctober 26, 2026 · arXiv
- MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile AgentsOctober 16, 2026 · arXiv
- MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency InjectionOctober 16, 2026 · arXiv
- Not All Graphs Are Maps: A Task-Agnostic Exploration Framework for Mobile GUI NavigationOctober 15, 2026 · arXiv
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer EnvironmentsOctober 14, 2026 · arXiv
- How GUI Agents Fail: GraceBench for Execution Stability DiagnosisOctober 14, 2026 · arXiv
- OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native ToolsOctober 9, 2026 · arXiv
- Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User InterfacesOctober 6, 2026 · arXiv
- Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?September 28, 2026 · Findings of EMNLP 2026
- MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory LearningSeptember 19, 2026 · USENIX Security 2026
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv
- PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI AgentsSeptember 12, 2026 · arXiv
- JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task CompositionSeptember 9, 2026 · arXiv
- APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI AgentsSeptember 7, 2026 · arXiv
- ElderBench: Benchmarking Autonomous Mobile Agents for Older AdultsSeptember 4, 2026 · arXiv
- GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent EnvironmentsAugust 30, 2026 · arXiv
- Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial EnvironmentsAugust 25, 2026 · arXiv
- ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across DevicesAugust 25, 2026 · arXiv
- GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data SynthesisAugust 24, 2026 · arXiv