Learning How to Remember: Memory Modality Selection for GUI Agents

Taeyun Roh , Suhyeong Park , Gijin Kim , Wonjune Jang , Jaewoo Kang

🏛 Institutions: Korea University , Myongji University , AIGEN Sciences
📅 Date: October 13, 2026
📑 Publisher: arXiv
💻 Env: Web Mobile Desktop
🔑 Keywords: memory modality SwitchMem GUI agent multimodal memory OSWorld Mind2Web

TLDR

Investigates memory representations in GUI agents and discovers that optimal memory modality (textual summary, action sequence, or screenshot embeddings) varies dynamically across task stages. Introduces SwitchMem, a policy that dynamically selects the memory modality for retrieved experience, boosting task success on OSWorld and Mind2Web.

Open paper arXiv Report issue