NVIDIA H100
By NVIDIA
The NVIDIA H100 is a data center GPU built on NVIDIA's Hopper architecture, designed specifically to accelerate training and inference for large-scale AI models. It has no display outputs and instead prioritizes high-bandwidth memory, fast…
Definition
The NVIDIA H100 is a data center GPU built on NVIDIA's Hopper architecture, designed specifically to accelerate training and inference for large-scale AI models. It has no display outputs and instead prioritizes high-bandwidth memory, fast inter-GPU interconnects, and specialized tensor cores tuned for the matrix multiplications underlying deep learning. The H100 is deployed in clusters of many units connected together, forming the computing backbone behind many large language model training runs and cloud AI inference services.
Overview
The H100 is NVIDIA's Hopper-generation successor to the widely used A100, purpose-built for the compute demands of modern AI rather than for graphics rendering. Where a consumer GPU like GeForce splits its silicon between shader cores for rendering and tensor cores for AI, the H100 devotes essentially its entire die to tensor cores and the memory system needed to feed them, since its only job is running the matrix and tensor operations that make up neural network training and inference. Mechanically, the H100 pairs its tensor cores with very high-bandwidth memory sitting physically close to the compute die, which matters because large model training is often limited by how fast data can move between memory and cores rather than by raw arithmetic speed. It supports reduced-precision numeric formats that trade some numerical accuracy for large speed and memory gains, a trade-off that works well for deep learning because neural networks tolerate that imprecision far better than most traditional numeric computing. NVIDIA's NVLink interconnect lets many H100s in a server exchange data directly at high speed, and further networking connects multiple servers into the large clusters that large language model training requires. Within NVIDIA's data center lineup, the H100 followed the A100 and was itself followed by the Blackwell-generation chips, each generation roughly doubling relevant AI throughput while also increasing power and cooling demands. Compared with consumer GeForce or professional RTX cards, the H100 drops ray-tracing hardware and display outputs entirely, and it commands a very different price and support tier aimed at enterprise and cloud customers rather than individual buyers. It also differs from more specialized AI accelerators from other vendors by offering broad compatibility with NVIDIA's CUDA software ecosystem, which most deep learning frameworks already target. In practice, the H100 is used by cloud providers, AI labs, and large enterprises to train and serve large language models, computer vision systems, and other deep learning workloads at scale, typically rented through cloud platforms rather than purchased individually given the cost and infrastructure required to operate them. Clusters of hundreds or thousands of H100s, connected by high-speed networking, are the kind of infrastructure behind training runs for today's largest AI models. The main limitations are cost, power consumption, and availability: H100 clusters require substantial capital investment, specialized data center cooling, and significant electricity, putting direct ownership out of reach for all but the largest organizations, while everyone else accesses the hardware through cloud rental. Reduced-precision formats can also introduce numerical trade-offs that require careful handling in training code. Teams with smaller-scale needs, or workloads that do not require frontier-scale model training, often use less expensive GPUs or the newer, more efficient Blackwell generation instead where available.
Key Features
- Built on NVIDIA's Hopper architecture for AI-specific compute
- No display outputs; entire die dedicated to AI compute
- High-bandwidth memory optimized for feeding large tensor operations
- Supports reduced-precision numeric formats for faster, more memory-efficient training
- NVLink interconnect enables high-speed communication between multiple H100 GPUs
- Deployed in large multi-GPU clusters for large language model training
- Successor to the A100 and predecessor to Blackwell-generation chips
- Broad compatibility with the CUDA deep learning software ecosystem