UIBert: Learning Generic Multimodal Representations for UI Understanding

Chongyang Bai , Xiaoxue Zang , Ying Xu , Srinivas Sunkara , Abhinav Rastogi , Jindong Chen , Blaise Agüera y Arcas

🏛 Institutions: Dartmouth College , Google Research
📅 Date: July 29, 2021
📑 Publisher: IJCAI 2021
💻 Env: Mobile
🔑 Keywords: model pretraining multimodal representation learning self-alignment UIBert

TLDR

UIBert is a transformer model for UI understanding trained with five UI-specific pretraining tasks over screenshots, text, and structural metadata. Its core idea is that the heterogeneous modalities inside a UI are self-aligned and can supervise one another to learn generic UI representations.

Open paper Report issue

Related papers (24)

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

April 13, 2026 · arXiv
SecAgent: Efficient Mobile GUI Agent with Semantic Context

March 9, 2026 · arXiv
Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

February 15, 2026 · arXiv
UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics

February 11, 2026 · arXiv
OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

January 28, 2026 · arXiv
MAI-UI Technical Report: Real-World Centric Foundation GUI Agents

December 26, 2025 · arXiv
Step-GUI Technical Report

December 17, 2025 · arXiv
AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent

November 30, 2025 · WACV 2026
ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

September 18, 2025 · ICLR 2026 (Oral)
Mobile-Agent-v3: Fundamental Agents for GUI Automation

August 21, 2025 · arXiv
UI-Venus Technical Report: Building High-performance UI Agents with RFT

August 14, 2025 · arXiv
AgentCPM‑GUI: Building Mobile‑Use Agents with Reinforcement Fine‑Tuning

June 2, 2025 · EMNLP 2025 System Demonstrations
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

May 27, 2025 · NeurIPS 2025 (Poster)
Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment

March 20, 2025 · MobiCom 2026
SpiritSight Agent: Advanced GUI Agent with One Look

March 5, 2025 · CVPR 2025 (Poster)
AppVLM: A Lightweight Vision Language Model for Online App Control

February 10, 2025 · arXiv
UI-TARS: Pioneering Automated GUI Interaction with Native Agents

January 21, 2025 · arXiv
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control

December 2, 2024 · Findings of ACL 2025
ShowUI: One Vision-Language-Action Model for GUI Visual Agent

November 26, 2024 · CVPR 2025 (Poster)
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

October 30, 2024 · ICLR 2025 (Spotlight)
AutoGLM: Autonomous Foundation Agents for GUIs

October 28, 2024 · arXiv
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding

September 23, 2024 · Findings of EMNLP 2024
MobileFlow: A Multimodal LLM for Mobile GUI Agent

July 5, 2024 · arXiv
CogAgent: A Visual Language Model for GUI Agents

December 14, 2023 · CVPR 2024 (Highlight)