A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants

This paper proposes an MCP-driven GUI architecture that lets existing applications expose navigation structure and action semantics to speech-enabled assistants through ViewModels and a GUI tree router. The design targets multimodal interaction with aligned spoken and visual feedback, and the paper also reports a small evaluation of locally deployable open-weight models for this setting.

• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
  February 3, 2026 · arXiv
• Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
  August 4, 2026 · arXiv
• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
  April 10, 2026 · arXiv
• MAI-UI Technical Report: Real-World Centric Foundation GUI Agents
  December 26, 2025 · arXiv
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  December 22, 2025 · arXiv
• Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents
  October 29, 2026 · arXiv
• Right Click, Wrong Box: Grading GUI Grounding by What a Click Does
  October 27, 2026 · arXiv
• Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
  October 27, 2026 · arXiv
• From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents
  October 27, 2026 · arXiv
• AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
  October 27, 2026 · arXiv
• GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents
  October 26, 2026 · arXiv
• PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
  October 24, 2026 · arXiv
• Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing
  October 24, 2026 · arXiv
• IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
  October 23, 2026 · arXiv
• Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier
  October 23, 2026 · arXiv
• Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training
  October 22, 2026 · arXiv
• PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
  October 22, 2026 · arXiv
• Who Verifies the Agent's Success Claim? Self-Report Leakage in GUI-Agent Completion Judging
  October 21, 2026 · arXiv
• Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
  October 20, 2026 · arXiv
• ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents
  October 19, 2026 · arXiv
• Focused Omni-Modal GUI Agents
  October 19, 2026 · arXiv
• Comparing Computer Use Behavior Across Multiple Levels of Abstraction
  October 19, 2026 · arXiv
• Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
  October 19, 2026 · arXiv
• Beyond Next-Screen Prediction: The Decision Value of GUI World Models
  October 17, 2026 · arXiv