Apache Hadoop
By Apache Software Foundation
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity hardware. It combines the Hadoop Distributed File System (HDFS) for storing data across many machines with…
Definition
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity hardware. It combines the Hadoop Distributed File System (HDFS) for storing data across many machines with the MapReduce programming model and the YARN resource manager for running parallel computations, letting organizations process petabyte-scale data reliably without specialized supercomputing hardware or proprietary storage appliances.
Overview
Hadoop was created to solve a problem that emerged as web-scale companies began collecting far more data than any single server could store or process economically. Rather than buying larger and more expensive machines, Hadoop's approach was to spread data and computation across many inexpensive commodity servers, accepting that individual machines would fail regularly and building fault tolerance into the software layer instead of relying on specialized hardware reliability. Mechanically, Hadoop's core consists of HDFS, which splits files into large blocks and replicates each block across multiple nodes so data survives individual machine failures, and YARN, which manages cluster compute resources and schedules jobs across available nodes. The original MapReduce engine processes data in two phases: a map phase that transforms and filters data in parallel across nodes close to where it is stored, and a reduce phase that aggregates the mapped results. This locality-aware design, moving computation to the data rather than the data to the computation, was central to Hadoop's ability to scale linearly by adding more nodes. Within the big-data ecosystem, Hadoop sits as foundational infrastructure that a wide array of other tools were built on top of or alongside, including Apache Hive for SQL-style querying, Apache HBase for random-access storage, and Apache Spark, which largely superseded raw MapReduce for processing because it keeps intermediate data in memory rather than writing to disk between stages, making it substantially faster for iterative workloads. Hadoop's HDFS and YARN layers, however, remain widely used underneath Spark and other engines. In practice, organizations use Hadoop clusters for batch processing of log data, building data lakes that retain raw data at low cost before it is transformed, and running large-scale ETL pipelines that feed downstream analytics and machine learning systems. Cloud providers offer managed Hadoop-ecosystem services that reduce the operational burden of running clusters directly, and many organizations have migrated workloads to cloud object storage paired with engines like Spark rather than running on-premises HDFS clusters. Hadoop's main limitations are its disk-based MapReduce processing model, which is slow for iterative or interactive workloads compared to in-memory engines, and the operational complexity of running and tuning a multi-node cluster. Many teams starting new big-data projects today choose Spark for processing and cloud object storage such as S3 for the storage layer, reserving classic Hadoop deployments for existing installations or scenarios that specifically need HDFS's data locality and replication model. Migrating off Hadoop entirely is nontrivial, though, since years of accumulated Hive tables, security policies, and job scheduling logic are often tightly coupled to the cluster, which is why large legacy Hadoop estates tend to persist for years after the industry's default shifted elsewhere.
Key Features
- HDFS distributes and replicates data blocks across cluster nodes for fault tolerance
- YARN manages cluster resources and schedules jobs across nodes
- MapReduce provides a parallel programming model for batch data processing
- Scales horizontally by adding commodity servers to a cluster
- Moves computation close to data to minimize network transfer
- Serves as the storage and resource foundation for tools like Hive and HBase
- Supports rack-awareness for optimizing replica placement across data centers
- Provides a mature ecosystem of connectors for ingestion and orchestration
Use Cases
Alternatives
Frequently Asked Questions
From the Blog
Introduction to Apache Spark for Beginners
Apache Spark is a fast, distributed engine for processing huge datasets across many machines. Learn what it is, how it works, and how to run your first job.
Read More ProgrammingWhat Is Hadoop? A Beginner's Guide to Big Data
Hadoop is an open-source framework that stores and processes very large datasets across many ordinary computers working together. This guide explains its core components, how it processes data, and when it still makes sense to use today.
Read More AI & TechnologyWhat Is PySpark? Python's Gateway to Big Data
PySpark is the Python API for Apache Spark, letting developers process massive datasets across many machines using familiar Python syntax. This guide covers what PySpark does, its core components, and when to reach for it.
Read More