Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations

This demo paper presents an interactive version of Sugilite in which users teach Android automation tasks through natural-language instructions plus GUI demonstrations. Its focus is learning reusable task procedures and concepts through conversational clarification grounded in app interfaces.

• PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
  August 30, 2019 · UIST 2019
• SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
  May 6, 2017 · CHI 2017
• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
  May 25, 2026 · arXiv
• ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
  April 13, 2026 · arXiv
• GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph
  January 24, 2026 · Journal of Intelligent Computing and Networking
• GUITester: Enabling GUI Agents for Exploratory Defect Discovery
  January 8, 2026 · arXiv
• Surfer 2: The Next Generation of Cross-Platform Computer Use Agents
  October 22, 2025 · arXiv
• CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs
  October 17, 2025 · NeurIPS 2025 (Poster)
• Agent-SAMA: State-Aware Mobile Assistant
  May 29, 2025 · AAAI 2026
• BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
  May 27, 2025 · EMNLP 2025 (Oral)
• GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
  May 22, 2025 · ACL 2025
• Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agent
  May 20, 2025 · arXiv
• ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
  April 30, 2025 · NAACL 2025 (Poster)
• MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation
  April 30, 2025 · NAACL 2025 (System Demonstrations)
• LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
  April 18, 2025 · arXiv
• DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents
  October 18, 2024 · ICLR 2025 (Poster)
• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
  October 9, 2024 · SIGDIAL 2025
• AppAgent v2: Advanced Agent for Flexible Mobile Interactions
  August 5, 2024 · arXiv
• MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices
  July 4, 2024 · arXiv
• AppAgent: Multimodal Agents as Smartphone Users
  December 21, 2023 · CHI 2025
• You Only Look at Screens: Multimodal Chain-of-Action Agents
  September 20, 2023 · Findings of ACL 2024
• AutoDroid: LLM-powered Task Automation in Android
  August 29, 2023 · MobiCom 2024
• META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
  May 23, 2022 · EMNLP 2022
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
  April 23, 2026 · arXiv