GUI Action Narrator: Where and When Did That Action Take Place?

Qinchen Wu, Difei Gao, Kevin Qinghong Lin, Zhuoyu Wu, Xiangwu Guo, Peiran Li, Weichen Zhang, Hengxu Wang, Mike Zheng Shou

🏛 Institutions: Show Lab, NUS, CAS, Shenzhen
📅 Date: June 19, 2024
📑 Publisher: arXiv
💻 Env: Desktop Web
🔑 Keywords: benchmark dataset Act2Cap GUI video captioning GUI Narrator

TLDR

GUI Action Narrator introduces Act2Cap, a benchmark and dataset of 4,189 GUI action video-captioning samples covering actions such as clicks, drags, and typing across desktop software and web tools. It also proposes GUI Narrator, which uses the cursor as a visual prompt plus temporal and spatial sampling to caption those actions more accurately than off-the-shelf multimodal models.

Open paper arXiv Edit on GitHub Report issue