GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

Shaoqing Zhang , Kehai Chen , Xuefeng Bai , Zhuosheng Zhang , Pengfei Zhang , Yang Xiang , Min Zhang

🏛 Institutions: HIT-Shenzhen , Pengcheng Laboratory , SJTU
📅 Date: September 28, 2026
📑 Publisher: arXiv
💻 Env: Mobile Web
🔑 Keywords: GUITAR failure analysis state transition graph offline evaluation AndroidControl Mind2Web

TLDR

GUITAR is a state-centric diagnostic framework that maps visually diverse screens to shared functional states in a State Transition Graph and analyzes GUI-agent failures over states and transitions. On AndroidControl and Mind2Web it finds that 60.4% of failures occur in 20% of states, and bottleneck-targeted guidance improves success rate.

Open paper Code Report issue