DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

A. Said Gurbuz , Ahmed Nassar , Sunghwan Hong , Marc Pollefeys , Peter W. J. Staar

🏛 Institutions: ETH , IBM Research , Microsoft
📅 Date: October 1, 2026
📑 Publisher: arXiv
💻 Env: Desktop
🔑 Keywords: DeskForge dataset model GUI grounding environment synthesis long-horizon tasks

TLDR

DeskForge is a controllable desktop environment that composes real applications under varied states, window layouts, appearances, and resolutions, and fuses screenshots, accessibility trees, and window geometry into dense element annotations. The resulting DeskForge-1M corpus (1.2M annotated observations) improves four fine-tuned vision-language models on five GUI grounding benchmarks and on the long-horizon WebArena-Infinity and OpenApps tasks.

Open paper arXiv Report issue