Vision Transformer (ViT)

Definition

Vision Transformer (ViT) is an image classification architecture introduced by Google Research in 2020 that applies the Transformer architecture — originally designed for text — directly to images, by splitting an image into fixed-size patches and treating each patch as a token in a sequence, rather than using convolutional layers.

Overview

Convolutional neural networks like ResNet had been the dominant architecture for computer vision for years, built around convolutional filters that exploit the local, grid-like structure of images. ViT took a very different approach, inspired by the Transformer's success in natural language processing: an image is split into a grid of fixed-size patches (for example, 16x16 pixels), each patch is flattened and linearly projected into an embedding, and the resulting sequence of patch embeddings — plus a learned positional embedding to preserve spatial information — is fed into a standard Transformer encoder, the same core architecture used in language models like BERT. The original ViT paper found that when trained on sufficiently large datasets, this patch-based Transformer approach could match or exceed the accuracy of state-of-the-art convolutional networks on image classification benchmarks, while requiring substantially fewer computational resources to train at the largest scales. Notably, ViT's advantage depended heavily on data scale: on smaller datasets, convolutional networks' built-in assumptions about image structure (locality, translation invariance) gave them an edge, but ViT's lack of these built-in biases became a strength once pretrained on very large datasets, allowing it to learn more flexible, general representations. ViT's success demonstrated that the Transformer architecture's core mechanism — self-attention, which lets every element in a sequence directly relate to every other element — was not specific to language, and it helped catalyze a broader trend toward unifying architectures across modalities. ViT-based image encoders are now widely used as components in multimodal systems, including CLIP's image encoder, and ViT variants (such as Swin Transformer and DeiT) have become common backbones in modern computer vision research and production systems alongside convolutional architectures.

Key Concepts

• Splits images into fixed-size patches treated as tokens, rather than using convolutions
• Applies a standard Transformer encoder architecture, shared with language models
• Uses learned positional embeddings to preserve spatial relationships between patches
• Matches or exceeds CNN accuracy when pretrained on large-scale datasets
• Requires large training data to outperform CNNs; less data-efficient at small scale
• Used as the image encoder inside multimodal models like CLIP

Use Cases

Frequently Asked Questions

Related in AI Models