AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Cheng Yang , Yifan Wu , Yutao Huang , Zhaohua Zhang , Beiduo Chen , Muxi Chen , Chenchen Zhao , Hexuan Deng , Haolin Yang , Geyuan Zhu , Sa Zhu , Jianhuan Zhuo , Qiuyong Xiao , Jianhao Ruan , Yiran Peng , Jiayi Zhang , Tian Ye , Xinlei Yu , Tianwen Jiang , Jihong Zhang , Yuyu Luo
- 🏛 Institutions
- Tencent Hunyuan
- 📅 Date
- October 1, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop
- 🔑 Keywords
TLDR
AutoGUIWorld synthesizes GUI interaction trajectories without running the corresponding software: a planner specifies atomic actions and their intended visual consequences, and an image generator iteratively edits the current screenshot to produce the next observation. The 79,266 step-level samples across Ubuntu, Windows, macOS, and Chrome raise Qwen3.5-35B-A3B from 33.0% to 40.8% on OSWorld and from 14.0% to 32.2% on ScienceBoard.
Related papers (24)
- WebWorld: A Large-Scale World Model for Web Agent TrainingFebruary 16, 2026 · arXiv
- UI-Oceanus: Scaling GUI Agents with Synthetic Environmental DynamicsFebruary 11, 2026 · arXiv
- Agent World Model in the Wild: Computer-Use Agent Environment SynthesisOctober 19, 2026 · arXiv
- OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using AgentsOctober 15, 2026 · arXiv
- Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient ExecutionSeptember 29, 2026 · arXiv
- HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use AgentsSeptember 29, 2026 · arXiv
- IntentScore: Intent-Conditioned Action Evaluation for Computer-Use AgentsApril 6, 2026 · arXiv
- EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic ExperienceJanuary 22, 2026 · arXiv
- ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use AgentsAugust 19, 2025 · ICLR 2026 (Poster)
- SEE: Structure-aware Exploring and Exploiting for Long-horizon GUI Agent Trajectory SynthesisJuly 20, 2026 · arXiv
- R-WoM: Retrieval-augmented World Model For Computer-use AgentsOctober 13, 2025 · ICLR 2026 (Poster)
- Evolving Agent InterfacesOctober 20, 2026 · arXiv
- CUA-Lite: An Open Platform for Computer Use AgentsOctober 19, 2026 · arXiv
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer UseOctober 15, 2026 · arXiv
- LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use AgentsOctober 15, 2026 · arXiv
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer EnvironmentsOctober 14, 2026 · arXiv
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon AgentsOctober 13, 2026 · arXiv
- Learning How to Remember: Memory Modality Selection for GUI AgentsOctober 13, 2026 · arXiv
- Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill InductionOctober 9, 2026 · arXiv
- Latent World Action Model for GUI AgentsOctober 9, 2026 · arXiv
- Right Screen, Wrong Transition: World Models as Verifiers for GUI AgentsOctober 8, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent TrainingOctober 5, 2026 · arXiv
- Control OSWorld: An AI Control Environment for GUI Computer Use AgentsOctober 2, 2026 · arXiv