Embedding

Definition

An embedding is a numeric vector representation of data — such as text, images, or audio — designed so that items with similar meaning or content are positioned close together in vector space.

Overview

Machine learning models operate on numbers, not raw text or pixels, so embeddings serve as the bridge between real-world data and a model's internal computations. An embedding model is trained so that semantically similar inputs — like the words "happy" and "joyful," or two similar product images — produce vectors that are close together by a distance metric such as cosine similarity, while dissimilar inputs produce vectors that are far apart. In large language models, tokens produced during tokenization are first converted into embedding vectors, which are then processed through layers using the attention mechanism to build increasingly rich, context-aware representations. Separately, standalone embedding models are widely used to convert entire documents, sentences, or images into a single vector for downstream use. These standalone embeddings are the foundation of vector search and semantic search: documents are embedded once and stored in a vector database, and a query embedding is compared against them to retrieve the most relevant matches — the core mechanism behind retrieval-augmented generation. The blog post How Large Language Models Actually Work explains this pipeline in more detail.

Key Concepts

• Represents data as a numeric vector capturing semantic meaning
• Similar items produce vectors that are close together in vector space
• Used internally by language models after tokenization
• Standalone embedding models convert documents or images into single vectors
• Foundation of vector search, semantic search, and RAG pipelines
• Distance metrics like cosine similarity measure embedding closeness

Use Cases

Frequently Asked Questions

Related in AI Fundamentals