ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

ASIL argues screenshot-and-click interaction is state-incomplete and brittle, and introduces an agent-native interaction layer exposing software via structured JSON observations and code-executable semantic actions, instantiated across 15 applications with a 300-task benchmark.

• CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
  September 4, 2026 · arXiv
• OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
  September 2, 2026 · arXiv
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
  April 23, 2026 · arXiv
• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
  April 10, 2026 · arXiv
• ShowUI-Aloha: Human-Taught GUI Agent
  January 12, 2026 · arXiv
• Surfer 2: The Next Generation of Cross-Platform Computer Use Agents
  October 22, 2025 · arXiv
• BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
  June 8, 2025 · ICML 2025 Workshop on Computer-use Agents
• LiteCUA: Computer as MCP Server for Computer-Use Agent on AIOS
  May 24, 2025 · arXiv
• UFO2: The Desktop AgentOS
  April 20, 2025 · arXiv
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  February 12, 2025 · arXiv
• PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World
  December 23, 2024 · arXiv
• AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
  October 24, 2024 · Findings of ACL 2025
• Agent S: An Open Agentic Framework that Uses Computers Like a Human
  October 10, 2024 · ICLR 2025 (Poster)
• AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
  September 26, 2024 · ACL 2025
• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
  February 12, 2024 · LLMAgents @ ICLR 2024
• Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
  June 13, 2023 · ICLR 2024 (Poster)
• Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions
  September 6, 2026 · arXiv
• Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents
  August 28, 2026 · arXiv
• AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction
  June 22, 2026 · arXiv
• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
  May 25, 2026 · arXiv
• ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
  May 24, 2026 · arXiv
• ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
  April 13, 2026 · arXiv
• OpeFlo: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding
  February 25, 2026 · arXiv
• LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent
  January 26, 2026 · ICLR 2026 (Poster)