Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

This work examines hybrid computer-use agents that can choose screenshots or text tools on OSWorld-MCP. It identifies an adoption gap in tool use and shows that training with a compatible post-tool observation policy can reduce image-context cost while improving the reported operating point.

• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
  April 10, 2026 · arXiv
• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents
  February 15, 2026 · arXiv
• MAI-UI Technical Report: Real-World Centric Foundation GUI Agents
  December 26, 2025 · arXiv
• The Amazing Agent Race: Strong Tool Users, Weak Navigators
  April 11, 2026 · arXiv
• The Tool Illusion: Rethinking Tool Use in Web Agents
  April 3, 2026 · arXiv
• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
  February 3, 2026 · arXiv
• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
  January 30, 2026 · arXiv
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  December 22, 2025 · arXiv
• A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants
  August 31, 2025 · arXiv
• TurkingBench: A Challenge Benchmark for Web Agents
  March 18, 2024 · NAACL 2025 (Oral)
• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
  September 9, 2026 · arXiv
• FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
  September 7, 2026 · arXiv
• AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents
  September 6, 2026 · arXiv
• From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents
  September 4, 2026 · arXiv
• CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
  September 4, 2026 · arXiv
• OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
  September 2, 2026 · arXiv
• Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
  September 1, 2026 · arXiv
• SIR: Self-improving Red-teaming for Compute Use Agents
  August 31, 2026 · arXiv
• CURA: Certified Runtime Alarms for Computer-Use Agents
  August 28, 2026 · arXiv
• ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
  August 27, 2026 · arXiv
• LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents
  August 26, 2026 · arXiv
• Reflection with Action-Induced Visual Differences for Desktop GUI Agents
  August 25, 2026 · arXiv
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
  August 25, 2026 · arXiv
• CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model Trajectories
  August 23, 2026 · arXiv