BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism

Qinzhuo Wu , Pengzhi Gao , Wei Liu , Jian Luan

🏛 Institutions: MiLM Plus , Xiaomi
📅 Date: May 27, 2025
📑 Publisher: EMNLP 2025 (Oral)
💻 Env: Mobile
🔑 Keywords: framework dataset error detection backtracking judgment reward BacktrackAgent

TLDR

BacktrackAgent addresses the lack of error recovery in mobile GUI agents by adding verifier, judger, and reflector modules plus an explicit backtracking mechanism. It also builds training data for judgment and reflection over post-action outcome pages, improving both task success and step accuracy on Mobile3M and Auto-UI.

Open paper arXiv Report issue