MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Haotian Zhang , Mingfei Gao , Zhe Gan , Philipp Dufter , Nina Wenzel , Forrest Huang , Dhruti Shah , Xianzhi Du , Bowen Zhang , Yanghao Li , Sam Dodge , Keen You , Zhen Yang , Aleksei Timofeev , Mingze Xu , Hong-You Chen , Jean-Philippe Fauconnier , Zhengfeng Lai , Haoxuan You , Zirui Wang , Afshin Dehghan , Peter Grasch , Yinfei Yang

🏛 Institutions: Apple
📅 Date: September 30, 2024
📑 Publisher: ICLR 2025 (Poster)
💻 Env
🔑 Keywords: model MM1.5 vision language model visual grounding reasoning data-centric analysis

TLDR

This paper introduces MM1.5, a family of multimodal large language models (MLLMs) ranging from 1B to 30B parameters, including dense and mixture-of-experts variants. MM1.5 enhances capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. The authors employ a data-centric training approach, utilizing high-quality OCR data and synthetic captions for continual pre-training, alongside an optimized visual instruction-tuning data mixture for supervised fine-tuning. Specialized variants, MM1.5-Video and MM1.5-UI, are designed for video understanding and mobile UI comprehension, respectively. Extensive empirical studies provide insights into the training processes, offering guidance for future MLLM development.

Open paper Report issue