Learn from Move: the Next Step for GUI Agents
Jiahao Lu , Yize Shen , Chuanyuan Tan , Mianqiu Huang , Shicheng Fang , Xinchi Chen , Yining Zheng , Yuxin Wang , Xipeng Qiu
- 🏛 Institutions
- Fudan , Shanghai Innovation Institute , Soochow University
- 📅 Date
- October 12, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop Web
- 🔑 Keywords
TLDR
Focuses on enabling GUI agents to autonomously explore and learn unfamiliar software interfaces through interactive trial and feedback. Introduces an exploration-driven learning protocol and benchmark where agents discover hidden menu actions and state semantics, improving generalization to unseen desktop applications.
Related papers (24)
- OSExpert: Computer-Use Agents Learning Professional Skills via ExplorationMarch 9, 2026 · arXiv
- Thinking vs. Doing: Agents that Reason by Scaling Test-Time InteractionJune 9, 2025 · SEA @ NeurIPS 2025 (Oral)
- OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and OptimizationOctober 25, 2024 · ACL 2025
- Towards Safe Autonomy for Computer UseOctober 13, 2026 · arXiv
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon AgentsOctober 13, 2026 · arXiv
- Learning How to Remember: Memory Modality Selection for GUI AgentsOctober 13, 2026 · arXiv
- Interaction-as-a-JudgeOctober 13, 2026 · arXiv
- Adaptive-Group Process Rewards for GUI AgentsOctober 13, 2026 · arXiv
- PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI AgentsOctober 12, 2026 · arXiv
- Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill InductionOctober 9, 2026 · arXiv
- Latent World Action Model for GUI AgentsOctober 9, 2026 · arXiv
- Right Screen, Wrong Transition: World Models as Verifiers for GUI AgentsOctober 8, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- Secure-CUA: Controlling Untrusted Influence in Computer-Use AgentsOctober 7, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent TrainingOctober 5, 2026 · arXiv
- TeleTune: Evolving Agent Skills From Offline TelemetryOctober 4, 2026 · arXiv
- CUAWright: A Minimal Unified Interface for Digital AgentsOctober 2, 2026 · arXiv
- Learning Reliable GUI Agents under Imperfect PriorsSeptember 30, 2026 · arXiv
- CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement LearningSeptember 26, 2026 · arXiv
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv
- MintAct: A Unified Visual Agent for Digital EnvironmentsSeptember 18, 2026 · arXiv
- LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI AgentsSeptember 9, 2026 · arXiv
- Qwen-CUA: Native Computer Use for (almost) EverythingAugust 3, 2026 · arXiv