A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants

This paper proposes an MCP-driven GUI architecture that lets existing applications expose navigation structure and action semantics to speech-enabled assistants through ViewModels and a GUI tree router. The design targets multimodal interaction with aligned spoken and visual feedback, and the paper also reports a small evaluation of locally deployable open-weight models for this setting.

• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
  February 3, 2026 · arXiv
• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
  April 10, 2026 · arXiv
• MAI-UI Technical Report: Real-World Centric Foundation GUI Agents
  December 26, 2025 · arXiv
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  December 22, 2025 · arXiv
• Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents
  June 12, 2026 · arXiv
• Demo2Tutorial: From Human Experience to Multimodal Software Tutorials
  June 2, 2026 · arXiv
• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models
  June 1, 2026 · arXiv
• GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
  May 29, 2026 · arXiv
• MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
  May 18, 2026 · arXiv
• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
  May 14, 2026 · arXiv
• Executable Agentic Memory for GUI Agent
  May 12, 2026 · arXiv
• LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning
  May 8, 2026 · arXiv
• Step-level Optimization for Efficient Computer-use Agents
  April 29, 2026 · arXiv
• Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
  April 28, 2026 · arXiv
• AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
  April 27, 2026 · arXiv
• Human-Guided Harm Recovery for Computer Use Agents
  April 20, 2026 · arXiv
• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
  April 15, 2026 · arXiv
• GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
  April 15, 2026 · arXiv
• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback
  April 14, 2026 · arXiv
• CocoaBench: Evaluating Unified Digital Agents in the Wild
  April 13, 2026 · arXiv
• Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
  April 9, 2026 · arXiv
• What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
  April 8, 2026 · Findings of ACL 2026
• MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs
  April 7, 2026 · arXiv
• GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
  April 6, 2026 · arXiv