EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

Xuetian Chen , Hangcheng Li , Jiaqing Liang , Sihang Jiang , Deqing Yang

🏛 Institutions: Fudan
📅 Date: October 25, 2024
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: dataset synthetic data GUI grounding multi-granularity data EDGE

TLDR

EDGE is a synthetic-data pipeline for GUI understanding that generates large-scale multi-granularity supervision from webpages. Models trained on the resulting dataset improve webpage understanding first and then transfer that gain to previously unseen desktop and mobile GUI environments with much less manual annotation.

Open paper Report issue