MMSkills: Multimodal Procedural Knowledge for Visual Agents

Introducing the MMSkills framework to strengthen the ability of artificial intelligence agents to make visual decisions. Unlike traditional technology packages that relied primarily on text or code, this system leverages multimodal procedural knowledge to combine text descriptions, status cards, and key frame images from multiple angles. Researchers propose an automatic generator that converts public interaction data into reusable technology, and a branch loading mechanism that provides only visual evidence needed for reasoning. Experiments have shown that this method consistently improves the performance of various models in operating system GUI control and game environments. As a result, these sources prove that external visual knowledge is complementary to pre-learning information within the model and suggest the development direction of the next generation of visual agents. https://arxiv.org/pdf/2605.13527