You Only Look at Screens: Multimodal Chain-of-Action Agents

Auto-GUI is a screenshot-only mobile GUI agent that avoids environment parsing and application-specific APIs. The paper introduces a chain-of-action prompting technique and evaluates the method on AITW, a device-control benchmark with 30K unique instructions.

• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
  May 25, 2026 · arXiv
• ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
  May 24, 2026 · arXiv
• GUITester: Enabling GUI Agents for Exploratory Defect Discovery
  January 8, 2026 · arXiv
• GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
  May 22, 2025 · ACL 2025
• LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
  April 18, 2025 · arXiv
• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
  October 9, 2024 · SIGDIAL 2025
• AutoDroid: LLM-powered Task Automation in Android
  August 29, 2023 · MobiCom 2024
• Android in the Wild: A Large-Scale Dataset for Android Device Control
  July 19, 2023 · NeurIPS 2023 Datasets and Benchmarks Track
• LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent
  January 26, 2026 · ICLR 2026 (Poster)
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  February 12, 2025 · arXiv
• WebWalker: Benchmarking LLMs in Web Traversal
  January 13, 2025 · arXiv
• The BrowserGym Ecosystem for Web Agent Research
  December 6, 2024 · TMLR
• SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
  May 30, 2023 · NeurIPS 2023
• Grounding Open-Domain Instructions to Automate Web Support Tasks
  March 30, 2021 · NAACL 2021
• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
  September 9, 2026 · arXiv
• APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  September 7, 2026 · arXiv
• Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions
  September 6, 2026 · arXiv
• ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults
  September 4, 2026 · arXiv
• GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
  August 30, 2026 · arXiv
• Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments
  August 25, 2026 · arXiv
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
  August 25, 2026 · arXiv
• Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  August 21, 2026 · arXiv
• OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
  July 30, 2026 · arXiv
• AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction
  June 22, 2026 · arXiv