Hadoop
By the Apache Software Foundation
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, using simple programming models to scale from a single machine to thousands of nodes.
Definition
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, using simple programming models to scale from a single machine to thousands of nodes.
Overview
Hadoop grew out of Google's early-2000s papers on the Google File System and MapReduce, and was created by Doug Cutting and Mike Cafarella as part of the Nutch search engine project before becoming its own Apache project in 2006. It answered a practical problem: how do you reliably store and process petabytes of data when no single machine can hold or crunch it all? Hadoop's answer was to split both storage and computation across many ordinary machines and assume that hardware will fail regularly, building redundancy into the system itself. At its core, Hadoop has two main pieces. The Hadoop Distributed File System (HDFS) breaks files into blocks and replicates them across multiple nodes so data survives disk and server failures. YARN (Yet Another Resource Negotiator) then schedules and manages compute jobs across the cluster, allowing the original MapReduce processing model to run alongside other engines. Over time, the surrounding ecosystem grew to include tools like Apache Hive for SQL-style querying and Apache Spark for much faster in-memory processing. Hadoop was the technology that made "big data" practical for mainstream enterprises in the 2010s, powering log analysis, data warehousing, and batch ETL pipelines at companies like Yahoo, Facebook, and countless others. As cloud data warehouses and faster engines like Spark matured, many organizations shifted workloads away from raw MapReduce, but HDFS-style distributed storage concepts and the broader Hadoop ecosystem remain foundational to how modern data platforms think about scale. Learners building data engineering skills often encounter Hadoop concepts before moving on to a course like SkillVeris's Apache Spark training.
Key Features
- HDFS distributed file system with automatic block replication for fault tolerance
- YARN resource manager for scheduling jobs across large clusters
- MapReduce programming model for parallel batch processing
- Designed to run on commodity hardware rather than specialized servers
- Horizontal scalability from a handful of nodes to thousands
- Rich ecosystem including Hive, Pig, HBase, and Sqoop for varied data workloads
- Data locality optimization that moves computation to where data resides
- Battle-tested at massive scale across web-era internet companies