OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

Qiushi Sun , Kanzhi Cheng , Zichen Ding , Chuanyang Jin , Yian Wang , Fangzhi Xu , Zhenyu Wu , Chengyou Jia , Liheng Chen , Zhoumianze Liu , Ben Kao , Guohao Li , Junxian He , Yu Qiao , Zhiyong Wu

🏛 Institutions: Shanghai AI Laboratory , HKU , JHU , SJTU , Oxford , HKUST
📅 Date: December 27, 2024
📑 Publisher: ACL 2025
💻 Env: General GUI
🔑 Keywords: dataset trajectory synthesis reverse task synthesis reward model OS-Genesis

TLDR

OS-Genesis tackles the lack of high-quality GUI trajectories by synthesizing them without preset tasks or human demonstrations. It first explores with step-level interactions, then retrospectively derives tasks and filters the resulting trajectories with a reward model, producing more diverse training data for GUI agents.

Open paper Report issue