What Is Hadoop?

Apache Hadoop is an open-source framework, originally created by Doug Cutting and Mike Cafarella and inspired by Google's GFS and MapReduce papers, that lets you store and process petabyte-scale datasets across clusters of ordinary servers. It bundles three core components: HDFS (Hadoop Distributed File System) for storage, YARN (Yet Another Resource Negotiator) for cluster resource management, and MapReduce as the original processing engine, all designed to tolerate hardware failure as a normal, expected event rather than an exception.

Before Hadoop, scaling data processing meant buying ever-bigger single machines (vertical scaling), which became prohibitively expensive and hit physical limits. Hadoop instead scales horizontally: add more commodity nodes to the cluster, and both storage capacity and processing power grow roughly linearly, at a fraction of the cost of specialized enterprise storage arrays.

Hadoop's defining idea is "move computation to the data" rather than moving data to computation — since network bandwidth is often the bottleneck at scale, code is shipped to the data node that already holds the relevant HDFS block, and results are aggregated afterward. Combined with 3x block replication by default, this gives Hadoop resilience: if a node dies mid-job, the scheduler reschedules the failed task on a node holding a replica, without operator intervention.