MM1
By Apple
MM1 is a family of multimodal large language models developed by Apple's AI research team, introduced in a March 2024 research paper as one of Apple's first public efforts describing large-scale vision-language model training, including a…
Definition
MM1 is a family of multimodal large language models developed by Apple's AI research team, introduced in a March 2024 research paper as one of Apple's first public efforts describing large-scale vision-language model training, including a mixture-of-experts variant.
Overview
MM1 was presented in a research paper titled "MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training," published by Apple researchers in March 2024. Rather than shipping MM1 as a consumer-facing product, Apple used the paper primarily to share empirical findings about what architectural and data choices matter most when pretraining multimodal models — for example, the paper found that image resolution and the number of image tokens fed to the model mattered more for performance than the specific choice of vision encoder architecture, and that a careful mix of image-caption data, interleaved image-text data, and text-only data produced the strongest results. The MM1 family ranged up to 30 billion dense parameters, and Apple also trained mixture-of-experts (MoE) variants reaching an effective capacity of up to 64 billion parameters while activating a smaller number of parameters per token, aiming for strong performance at a manageable inference cost. MM1 was evaluated on standard multimodal benchmarks including visual question answering and image captioning tasks, and Apple reported competitive few-shot and in-context learning capabilities compared to contemporaneous multimodal models. Unlike open releases such as Idefics2 or LLaVA, Apple did not release MM1's weights publicly; the research served primarily to inform the company's internal multimodal AI development, which fed into Apple's broader on-device and cloud AI strategy under Apple Intelligence, announced later in 2024. Apple has continued this research line with later papers (including MM1.5), refining data curation and training strategies for multimodal understanding, positioning MM1 as an important research contribution to the field's understanding of multimodal pretraining even without a corresponding public product release.
Key Concepts
- Detailed research paper on multimodal LLM pretraining methodology and findings
- Dense models up to 30 billion parameters, plus mixture-of-experts variants up to 64B effective capacity
- Empirical findings on the importance of image resolution and token count over encoder choice
- Careful data mixture of captioned images, interleaved image-text, and text-only data
- Evaluated on standard visual question answering and captioning benchmarks
- Weights not released publicly; used to inform Apple's internal AI development
- Followed by later research iterations such as MM1.5