Kosmos
Kosmos is a series of multimodal large language models developed by Microsoft Research, designed to jointly perceive, ground, and reason across text and images (and, in later versions, additional modalities) within a single model.
Definition
Kosmos is a series of multimodal large language models developed by Microsoft Research, designed to jointly perceive, ground, and reason across text and images (and, in later versions, additional modalities) within a single model.
Overview
Kosmos-1, introduced by Microsoft Research in 2023, was presented as a "Multimodal Large Language Model" (MLLM) capable of perceiving general modalities, following instructions, and performing in-context learning across both text and visual inputs, positioning it as a step toward models that combine language capabilities with genuine perception rather than treating vision as a bolt-on feature. Kosmos-1 could handle tasks such as visual question answering, image captioning, and multimodal dialogue, and it introduced early evaluation of "Raven IQ test"-style nonverbal reasoning to probe multimodal reasoning capability beyond typical benchmarks. Microsoft followed with Kosmos-2 in 2023, which added grounding capability — the ability to link noun phrases and referring expressions in generated text to specific spatial regions in an image, represented as bounding boxes. This grounding lets Kosmos-2 produce descriptions and answers that are spatially tied to actual objects in an image, supporting tasks like referring expression comprehension and grounded visual question answering, where the model must point to exactly what it's talking about rather than just describing an image in the abstract. The Kosmos line reflects Microsoft's broader multimodal research direction, alongside related efforts, and its grounding and perception techniques have informed subsequent multimodal model research across the industry. While Kosmos itself is primarily a research model rather than a widely deployed commercial product, its architecture and training methodology contributed to the broader push toward unified multimodal foundation models that can perceive, ground, and reason jointly across text and images, a category that later included models like GPT-4V, Gemini, and LLaVA.
Key Concepts
- Multimodal large language model jointly processing text and images
- Kosmos-2 introduces grounding, linking text to spatial regions via bounding boxes
- Supports visual question answering and image captioning
- In-context learning across multimodal inputs
- Evaluated on nonverbal reasoning benchmarks like Raven IQ-style tests
- Developed by Microsoft Research as part of broader multimodal AI research