Falcon-UI: Understanding GUI Before Following User Instructions

Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, Xiangyang Ji

🏛 Institutions: Institute of Information Engineering, CAS, Nankai University, Tsinghua, Beijing Academy of Artificial Intelligence, University of Chinese Academy of Sciences
📅 Date: December 12, 2024
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: model dataset Insight-UI Dataset GUI understanding instruction-free pretraining Falcon-UI

TLDR

Falcon-UI studies whether GUI-context understanding should be learned before instruction following. It introduces the large instruction-free Insight-UI Dataset for GUI pretraining and shows that this staged training improves a 7B model enough to approach much larger baselines.

Open paper arXiv Edit on GitHub Report issue