You Only Look at Screens: Multimodal Chain-of-Action Agents

Auto-GUI is a screenshot-only mobile GUI agent that avoids environment parsing and application-specific APIs. The paper introduces a chain-of-action prompting technique and evaluates the method on AITW, a device-control benchmark with 30K unique instructions.

• Not All Graphs Are Maps: A Task-Agnostic Exploration Framework for Mobile GUI Navigation
  October 15, 2026 · arXiv
• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
  May 25, 2026 · arXiv
• ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
  May 24, 2026 · arXiv
• GUITester: Enabling GUI Agents for Exploratory Defect Discovery
  January 8, 2026 · arXiv
• GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
  May 22, 2025 · ACL 2025
• LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
  April 18, 2025 · arXiv
• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
  October 9, 2024 · SIGDIAL 2025
• AutoDroid: LLM-powered Task Automation in Android
  August 29, 2023 · MobiCom 2024
• Android in the Wild: A Large-Scale Dataset for Android Device Control
  July 19, 2023 · NeurIPS 2023 Datasets and Benchmarks Track
• A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution
  October 11, 2026 · arXiv
• LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent
  January 26, 2026 · ICLR 2026 (Poster)
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  February 12, 2025 · arXiv
• WebWalker: Benchmarking LLMs in Web Traversal
  January 13, 2025 · arXiv
• The BrowserGym Ecosystem for Web Agent Research
  December 6, 2024 · TMLR
• SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
  May 30, 2023 · NeurIPS 2023
• Grounding Open-Domain Instructions to Automate Web Support Tasks
  March 30, 2021 · NAACL 2021
• iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents
  October 26, 2026 · arXiv
• AndroidFlux: Evaluating Failure Recovery and Recovery Action Reward Modeling in Mobile-Use Multimodal Agents
  October 25, 2026 · arXiv
• MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile Agents
  October 16, 2026 · arXiv
• MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection
  October 16, 2026 · arXiv
• MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
  October 14, 2026 · arXiv
• How GUI Agents Fail: GraceBench for Execution Stability Diagnosis
  October 14, 2026 · arXiv
• OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools
  October 9, 2026 · arXiv
• Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces
  October 6, 2026 · arXiv