Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Satwik Ram Kodandaram , Monalika Padma Reddy , Xiaojun Bi , Jiawei Zhou , I. V. Ramakrishnan , Vikas Ashok
- 🏛 Institutions
- Stony Brook University , Old Dominion University
- 📅 Date
- September 1, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop
- 🔑 Keywords
TLDR
A three-week diary study with 8 blind users of OLLA, a screen-reader-accessible computer-use-agent prototype, collects 1,258 commands across 12 applications; GPT-5 reaches 52.5% success, with a failure taxonomy spanning grounding, planning, constraint tracking, and termination.
Related papers (24)
- ElderBench: Benchmarking Autonomous Mobile Agents for Older AdultsSeptember 4, 2026 · arXiv
- Widget Captioning: Generating Natural Language Description for Mobile User Interface ElementsNovember 30, 2020 · EMNLP 2020
- JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task CompositionSeptember 9, 2026 · arXiv
- FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?September 7, 2026 · arXiv
- AgentHijack: Visual Patch Attacks on Multimodal Computer-Use AgentsSeptember 6, 2026 · arXiv
- From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use AgentsSeptember 4, 2026 · arXiv
- CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI AgentsSeptember 4, 2026 · arXiv
- OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human DemonstrationsSeptember 2, 2026 · arXiv
- SIR: Self-improving Red-teaming for Compute Use AgentsAugust 31, 2026 · arXiv
- CURA: Certified Runtime Alarms for Computer-Use AgentsAugust 28, 2026 · arXiv
- ASIL: Replacing Screenshot-and-Click with Structured State and Semantic ActionsAugust 27, 2026 · arXiv
- LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI AgentsAugust 26, 2026 · arXiv
- Reflection with Action-Induced Visual Differences for Desktop GUI AgentsAugust 25, 2026 · arXiv
- ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across DevicesAugust 25, 2026 · arXiv
- CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model TrajectoriesAugust 23, 2026 · arXiv
- Spine-Branch Coordination for Multi-agent Computer UseAugust 22, 2026 · arXiv
- Inducing Task Models from Computer-Use TracesAugust 20, 2026 · arXiv
- Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use AgentsAugust 4, 2026 · arXiv
- Qwen-CUA: Native Computer Use for (almost) EverythingAugust 3, 2026 · arXiv
- MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action DistillationJuly 31, 2026 · arXiv
- CUADebug: Diagnosing and Repairing Computer-Use Agent FailuresJuly 31, 2026 · arXiv
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI AgentsJuly 30, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv
- OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic GroundingJuly 29, 2026 · arXiv