Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

Alice Gao , Weixi Tong , Rishab Vempati , Katharina Reinecke , R. Benjamin Shapiro , Tianyi Zhang , Jason Wu

🏛 Institutions: UW , Purdue University
📅 Date: April 28, 2026
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: model usability interaction flow prioritization uxCUA

TLDR

uxCUA frames automated GUI usability assessment as a learned agent task by (i) prioritizing important interaction flows, (ii) executing them with human-like interactions, and (iii) predicting a numerical usability score. The trained agent outperforms larger general-purpose LLMs at scoring usability and generating actionable critiques on both experimental and production interfaces.

Open paper Report issue