A Data-Driven Approach for Learning to Control Computers

Peter C. Humphreys , David Raposo , Tobias Pohlen , Gregory Thornton , Rachita Chhaparia , Alistair Muldal , Josh Abramson , Petko Georgiev , Alex Goldin , Adam Santoro , Timothy Lillicrap

🏛 Institutions: Google DeepMind
📅 Date: February 16, 2022
📑 Publisher: ICML 2022
💻 Env: Desktop
🔑 Keywords: computer control behavioral cloning reinforcement learning demonstration data MiniWoB++

TLDR

Studies computer control as a data-driven learning problem using natural-language goals plus low-level mouse and keyboard actions. The work combines human demonstrations with RL-style training and shows strong cross-task generalization on desktop-style control tasks.

Open paper Report issue

Related papers (24)

Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation

September 28, 2025 · arXiv
ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents

August 19, 2025 · ICLR 2026 (Poster)
DPO Learning with LLMs-Judge Signal for Computer Use Agents

June 3, 2025 · arXiv
ZeroGUI: Automating Online GUI Learning at Zero Human Cost

May 29, 2025 · arXiv
ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay

May 22, 2025 · arXiv
GUI-R1: A Generalist R1-Style Vision-Language Action Model for GUI Agents

April 14, 2025 · arXiv
ScreenAgent: A Vision Language Model-driven Computer Control Agent

February 13, 2024 · IJCAI 2024
Language Models can Solve Computer Tasks

March 30, 2023 · NeurIPS 2023
GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

May 29, 2026 · arXiv
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

May 25, 2026 · arXiv
LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning

May 8, 2026 · arXiv
ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

April 13, 2026 · arXiv
Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions

April 8, 2026 · arXiv
Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

April 7, 2026 · ACL 2026
WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale

March 2026 · Blog Post
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience

March 25, 2026 · arXiv
OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

March 19, 2026 · arXiv
Generalization in Online Reinforcement Learning for Mobile Agents

March 8, 2026 · arXiv
WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents

March 5, 2026 · arXiv
CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning

March 3, 2026 · arXiv
GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

February 25, 2026 · arXiv
OpAgent: Operator Agent for Web Navigation

February 14, 2026 · arXiv
Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

February 14, 2026 · arXiv
Adaptive Milestone Reward for GUI Agents

February 12, 2026 · arXiv