Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Yang Li , Gang Li , Luheng He , Jingjie Zheng , Hong Li , Zhiwei Guan

🏛 Institutions: Google Research , Georgia Tech
📅 Date: November 30, 2020
📑 Publisher: EMNLP 2020
💻 Env: Mobile
🔑 Keywords: dataset benchmark WidgetCaption accessibility widget captioning multimodal generation

TLDR

This paper formulates widget captioning as generating natural-language descriptions for mobile UI elements from screenshot and structural input. It contributes a large dataset with 162,859 phrases for 61,285 UI elements and positions caption generation as a foundation for accessibility and language-based UI interaction.

Open paper Report issue