Hadoop vs Spark

Hadoop MapReduce and Apache Spark are often framed as competitors, but they are better understood as two execution engines that can both run on the same storage and resource layers. MapReduce processes data in discrete map and reduce phases, writing intermediate results to disk between stages, which makes it extremely fault-tolerant but slow for workloads that revisit the same data repeatedly. Spark instead builds a directed acyclic graph (DAG) of transformations over Resilient Distributed Datasets (RDDs) and can cache data in memory across stages, trading some fault-tolerance simplicity for substantially higher throughput on iterative and interactive workloads.

In classic MapReduce, each job runs map tasks that emit key-value pairs, a shuffle-and-sort phase partitions and sorts those pairs across reducers, and reduce tasks aggregate the results — with intermediate output persisted to local disk at every stage boundary. Spark's engine instead compiles a chain of transformations (map, filter, join, reduceByKey) into a DAG of stages split at shuffle boundaries, executes them lazily only when an action like collect() or save() is called, and can keep intermediate RDDs or DataFrames cached in memory (via .persist() or .cache()) across multiple actions, avoiding redundant disk reads for iterative algorithms.

For a single-pass batch ETL job over a huge dataset, MapReduce's disk-based checkpointing between stages provides strong fault tolerance at a modest performance cost. But for iterative algorithms — logistic regression via gradient descent, PageRank, or repeated graph traversals — MapReduce must reread and rewrite the entire dataset from HDFS at every iteration, whereas Spark can cache the working set in memory once and reuse it, commonly delivering 10–100x speedups on benchmarks like the original Spark paper's logistic regression comparison.

The Hadoop ecosystem grew around MapReduce with tools like Hive (SQL over MapReduce or Tez), Pig (dataflow scripting), and HBase (wide-column store), all designed for large-scale batch processing with strong durability guarantees. Spark's ecosystem — Spark SQL, Structured Streaming, MLlib, and GraphX — targets a broader range of workloads including near-real-time streaming and iterative machine learning, unified under a single programming model in Scala, Python, or Java. In practice, many organizations run both: MapReduce or Spark for heavy nightly batch ETL where cost-per-terabyte matters most, and Spark for interactive analytics, streaming pipelines, and ML training where latency and iteration speed matter most.