GUI Agents Papers
Star · 821

Falcon-UI: Understanding GUI Before Following User Instructions

Huawen Shen , Chang Liu , Gengluo Li , Xinlong Wang , Yu Zhou , Can Ma , Xiangyang Ji

🏛 Institutions
Institute of Information Engineering , CAS , Nankai University , Tsinghua , Beijing Academy of Artificial Intelligence , University of Chinese Academy of Sciences
📅 Date
December 12, 2024
📑 Publisher
arXiv
💻 Env
General GUI
🔑 Keywords
modeldatasetInsight-UI DatasetGUI understandinginstruction-free pretrainingFalcon-UI
TLDR

Falcon-UI studies whether GUI-context understanding should be learned before instruction following. It introduces the large instruction-free Insight-UI Dataset for GUI pretraining and shows that this staged training improves a 7B model enough to approach much larger baselines.

Open paper arXiv Report issue
Related papers (24)