ScaleTrack: Scaling and back-tracking Automated GUI Agents

Jing Huang , Zhixiong Zeng , Wenkang Han , Yufeng Zhong , Liming Zheng , Shuai Fu , Jingyuan Chen , Lin Ma

🏛 Institutions: Meituan , ZJU , University of Adelaide
📅 Date: May 1, 2025
📑 Publisher: arXiv
💻 Env: Desktop Mobile Web
🔑 Keywords: GUI grounding backtracking planning data scaling historical action backtracking ScaleTrack

TLDR

ScaleTrack targets two training bottlenecks in automated GUI agents: weak grounding data coverage and the lack of backtracking behavior during planning. It aggregates GUI samples from heterogeneous sources into a unified grounding corpus and trains agents to predict the next action together with the historical actions that led to the current screen.

Open paper Report issue