GUI Agents with Foundation Models: A Comprehensive Survey

Shuai Wang , Weiwen Liu , Jingxuan Chen , Yuqi Zhou , Weinan Gan , Xingshan Zeng , Yuhan Che , Shuai Yu , Xinlong Hao , Kun Shao , Bin Wang , Chuhan Wu , Yasheng Wang , Ruiming Tang , Jianye Hao

🏛 Institutions: Huawei Noah's Ark Lab
📅 Date: November 7, 2024
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: survey foundation models taxonomy industrial applications

TLDR

This survey organizes foundation-model GUI agents around data resources, agent construction, taxonomy, and industrial applications. It also summarizes open challenges around the benchmark-reality gap, agent self-evolution, and inference efficiency.

Open paper Report issue

Related papers (24)

Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

November 17, 2024 · arXiv
Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization

September 2, 2026 · arXiv
Software Engineering for and with GUI Agent

August 10, 2026 · arXiv
How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction

February 12, 2026 · arXiv
A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?

May 16, 2025 · arXiv
A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning

April 29, 2025 · arXiv
Towards Trustworthy GUI Agents: A Survey

March 30, 2025 · arXiv
OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use

December 20, 2024 · ACL 2025
GUI Agents: A Survey

December 18, 2024 · Findings of ACL 2025
Mapping the Design Space of User Experience for Computer Use Agents

February 7, 2026 · IUI 2026
LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects

April 28, 2025 · TMLR 2025
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models

March 30, 2025 · KDD 2025
WebSuite: Systematically Evaluating Why Web Agents Fail

June 1, 2024 · arXiv
Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents

October 29, 2026 · arXiv
Right Click, Wrong Box: Grading GUI Grounding by What a Click Does

October 27, 2026 · arXiv
Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

October 27, 2026 · arXiv
From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

October 27, 2026 · arXiv
AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

October 27, 2026 · arXiv
GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

October 26, 2026 · arXiv
PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

October 24, 2026 · arXiv
Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing

October 24, 2026 · arXiv
IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

October 23, 2026 · arXiv
Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier

October 23, 2026 · arXiv
Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training

October 22, 2026 · arXiv