ScaleTrack: Scaling and back-tracking Automated GUI Agents

Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, Lin Ma

🏛 Institutions: Meituan, ZJU, University of Adelaide
📅 Date: May 1, 2025
📑 Publisher: arXiv
💻 Env: Desktop Mobile Web
🔑 Keywords: GUI grounding backtracking planning data scaling historical action backtracking ScaleTrack

TLDR

ScaleTrack targets two training bottlenecks in automated GUI agents: weak grounding data coverage and the lack of backtracking behavior during planning. It aggregates GUI samples from heterogeneous sources into a unified grounding corpus and trains agents to predict the next action together with the historical actions that led to the current screen.

Open paper arXiv Edit on GitHub Report issue