Programming with Pixels: Can Computer-Use Agents do Software Engineering?

This paper introduces Programming with Pixels, a visual IDE environment for evaluating whether generalist computer-use agents can handle software engineering tasks rather than only simple desktop or web interactions. It also presents PwP-Bench, a benchmark spanning 15 software-engineering tasks across languages and modalities. The results show that purely visual computer-use agents lag behind specialist coding agents, but narrow text APIs such as file editing and bash dramatically narrow that gap.

• WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
  April 30, 2026 · arXiv
• The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
  April 12, 2026 · arXiv
• HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
  April 10, 2026 · arXiv
• Gym-Anything: Turn any Software into an Agent Environment
  April 7, 2026 · arXiv
• HippoCamp: Benchmarking Contextual Agents on Personal Computers
  April 1, 2026 · arXiv
• PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents
  March 9, 2026 · arXiv