A Data-Driven Approach for Learning to Control Computers

Peter C. Humphreys , David Raposo , Tobias Pohlen , Gregory Thornton , Rachita Chhaparia , Alistair Muldal , Josh Abramson , Petko Georgiev , Alex Goldin , Adam Santoro , Timothy Lillicrap

🏛 Institutions: Google DeepMind
📅 Date: February 16, 2022
📑 Publisher: ICML 2022
💻 Env: Desktop
🔑 Keywords: computer control behavioral cloning reinforcement learning demonstration data MiniWoB++

TLDR

Studies computer control as a data-driven learning problem using natural-language goals plus low-level mouse and keyboard actions. The work combines human demonstrations with RL-style training and shows strong cross-task generalization on desktop-style control tasks.

Open paper Publisher Report issue

Related papers (24)

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

July 25, 2026 · arXiv
SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL

July 13, 2026 · arXiv
STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization

November 17, 2025 · Findings of ACL 2026
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation

September 28, 2025 · arXiv
ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents

August 19, 2025 · ICLR 2026 (Poster)
DPO Learning with LLMs-Judge Signal for Computer Use Agents

June 3, 2025 · arXiv
ZeroGUI: Automating Online GUI Learning at Zero Human Cost

May 29, 2025 · arXiv
ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay

May 22, 2025 · arXiv
GUI-R1: A Generalist R1-Style Vision-Language Action Model for GUI Agents

April 14, 2025 · arXiv
ScreenAgent: A Vision Language Model-driven Computer Control Agent

February 13, 2024 · IJCAI 2024
Language Models can Solve Computer Tasks

March 30, 2023 · NeurIPS 2023
WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

August 27, 2026 · arXiv
Task-Adaptive Rubrics for GUI Reward Modeling

August 25, 2026 · arXiv
Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

August 25, 2026 · arXiv
GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

August 24, 2026 · arXiv
Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents

August 22, 2026 · arXiv
GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

May 29, 2026 · arXiv
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

May 25, 2026 · arXiv
LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning

May 8, 2026 · arXiv
ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

April 13, 2026 · arXiv
Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions

April 8, 2026 · arXiv
Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

April 7, 2026 · ACL 2026
WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale

March 2026 · Blog Post
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience

March 25, 2026 · arXiv