GUI Agents: A Survey

Dang Nguyen , Jian Chen , Yu Wang , Gang Wu , Namyong Park , Zhengmian Hu , Hanjia Lyu , Junda Wu , Ryan Aponte , Yu Xia , Xintong Li , Jing Shi , Hongjie Chen , Viet Dac Lai , Zhouhang Xie , Sungchul Kim , Ruiyi Zhang , Tong Yu , Mehrab Tanjim , Nesreen K. Ahmed , Puneet Mathur , Seunghyun Yoon , Lina Yao , Branislav Kveton , Jihyung Kil , Thien Huu Nguyen , Trung Bui , Tianyi Zhou , Ryan A. Rossi , Franck Dernoncourt

🏛 Institutions: UMD , State University of New York at Buffalo , University of Oregon , Adobe Research , University of Rochester , UC San Diego , CMU , Dolby Labs , Cisco Research , University of New South Wales
📅 Date: December 18, 2024
📑 Publisher: Findings of ACL 2025
💻 Env: General GUI
🔑 Keywords: survey benchmarks architectures training evaluation

TLDR

This survey organizes GUI-agent research around benchmarks, evaluation metrics, architectures, and training methods for agents powered by large foundation models. It proposes a unified perception-reasoning-planning-acting framework and highlights the open problems that remain across the stack.

Open paper arXiv Publisher Code Report issue