UIBert: Learning Generic Multimodal Representations for UI Understanding
Chongyang Bai , Xiaoxue Zang , Ying Xu , Srinivas Sunkara , Abhinav Rastogi , Jindong Chen , Blaise Agüera y Arcas
- 🏛 Institutions
- Dartmouth College , Google Research
- 📅 Date
- July 29, 2021
- 📑 Publisher
- IJCAI 2021
- 💻 Env
- Mobile
- 🔑 Keywords
TLDR
UIBert is a transformer model for UI understanding trained with five UI-specific pretraining tasks over screenshots, text, and structural metadata. Its core idea is that the heterogeneous modalities inside a UI are self-aligned and can supervise one another to learn generic UI representations.
Related papers (24)
- Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI AgentsSeptember 23, 2026 · arXiv
- Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory PredictionSeptember 22, 2026 · EMNLP 2026
- MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory LearningSeptember 19, 2026 · USENIX Security 2026
- MintAct: A Unified Visual Agent for Digital EnvironmentsSeptember 18, 2026 · arXiv
- BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI AgentsSeptember 11, 2026 · arXiv
- LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI AgentsSeptember 9, 2026 · arXiv
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI AgentsJuly 30, 2026 · arXiv
- Scaling GUI Agents with Visual State TransitionsJuly 27, 2026 · arXiv
- Xiaomi-GUI-0 Technical ReportJune 30, 2026 · arXiv
- ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI AgentsApril 13, 2026 · arXiv
- SecAgent: Efficient Mobile GUI Agent with Semantic ContextMarch 9, 2026 · arXiv
- CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities ReasoningFebruary 27, 2026 · ICML 2026 (Poster)
- Mobile-Agent-v3.5: Multi-platform Fundamental GUI AgentsFebruary 15, 2026 · arXiv
- UI-Oceanus: Scaling GUI Agents with Synthetic Environmental DynamicsFebruary 11, 2026 · arXiv
- OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task ExecutionJanuary 28, 2026 · arXiv
- MAI-UI Technical Report: Real-World Centric Foundation GUI AgentsDecember 26, 2025 · arXiv
- Step-GUI Technical ReportDecember 17, 2025 · arXiv
- AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agentNovember 30, 2025 · WACV 2026
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform DataSeptember 18, 2025 · ICLR 2026 (Oral)
- Mobile-Agent-v3: Fundamental Agents for GUI AutomationAugust 21, 2025 · arXiv
- UI-Venus Technical Report: Building High-performance UI Agents with RFTAugust 14, 2025 · arXiv
- AgentCPM‑GUI: Building Mobile‑Use Agents with Reinforcement Fine‑TuningJune 2, 2025 · EMNLP 2025 System Demonstrations
- UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI AgentsMay 27, 2025 · NeurIPS 2025 (Poster)
- Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical DeploymentMarch 20, 2025 · MobiCom 2026