You Only Look at Screens: Multimodal Chain-of-Action Agents

Zhuosheng Zhang, Aston Zhang

🏛 Institutions: SJTU, Meta
📅 Date: September 20, 2023
📑 Publisher: Findings of ACL 2024
💻 Env: Mobile
🔑 Keywords: framework benchmark chain-of-action Auto-GUI AITW screenshot-only control

TLDR

Auto-GUI is a screenshot-only mobile GUI agent that avoids environment parsing and application-specific APIs. The paper introduces a chain-of-action prompting technique and evaluates the method on AITW, a device-control benchmark with 30K unique instructions.

Open paper Edit on GitHub Report issue

Related papers

GUITester: Enabling GUI Agents for Exploratory Defect Discovery

January 8, 2026 · arXiv
GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent

May 22, 2025 · ACL 2025
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark

April 18, 2025 · arXiv
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

October 9, 2024 · SIGDIAL 2025
AutoDroid: LLM-powered Task Automation in Android

August 29, 2023 · MobiCom 2024
Android in the Wild: A Large-Scale Dataset for Android Device Control

July 19, 2023 · NeurIPS 2023 Datasets and Benchmarks Track