What Is a Resilient Distributed Dataset?
A Resilient Distributed Dataset (RDD) is Spark's original low-level abstraction for a distributed, immutable collection of objects that is automatically split into partitions and spread across the executors in a cluster. Because RDDs are immutable, every transformation produces a brand-new RDD rather than mutating data in place, which makes parallel execution safe without locks.
Cricket analogy: Think of an RDD like a Test squad of 15 players split across three warm-up nets (partitions); each net trains independently, and no player secretly swaps himself for another mid-drill because the original squad list is fixed.
Creating RDDs
RDDs are created in two main ways: by parallelizing an existing in-memory collection with sc.parallelize(), which is useful for testing and small datasets, or by loading external data with sc.textFile() or hadoopFile(), which reads files from HDFS, S3, or the local filesystem and automatically partitions them, typically one partition per HDFS block.
Cricket analogy: It's like a captain either hand-picking a practice squad from players already in the dressing room (parallelize) or calling up fresh talent from the domestic circuit by reading through state team rosters (textFile).
from pyspark import SparkContext
sc = SparkContext(appName="RDDBasics")
# Create an RDD from an in-memory Python list
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6], numSlices=3)
print(numbers_rdd.getNumPartitions()) # 3
# Create an RDD by reading a text file (e.g., from HDFS or local disk)
lines_rdd = sc.textFile("hdfs:///data/server_logs/*.log")
print(lines_rdd.getNumPartitions()) # roughly one partition per HDFS block
# Both are RDDs and support the same transformation/action API
error_lines = lines_rdd.filter(lambda line: "ERROR" in line)
print(error_lines.count())
Fault Tolerance Through Lineage
RDDs achieve fault tolerance not by replicating data across nodes but by recording a lineage graph, a DAG of the transformations used to build the RDD from its original source data. If a partition is lost because an executor crashes, Spark simply recomputes only that lost partition by replaying the recorded transformations, instead of restoring from a full replica.
Cricket analogy: It's like a scorer who doesn't keep a duplicate scoreboard for every over, but instead keeps the ball-by-ball commentary log, so if one page of the scoresheet is torn, the final score can be reconstructed from the log.
RDDs are still the foundation Spark is built on, but for most application code today you should reach for the DataFrame or Dataset API instead, since it offers the same fault-tolerant lineage model plus Catalyst query optimization and Tungsten memory management, which plain RDDs do not get automatically.
Partitioning and Persistence
The number and layout of partitions in an RDD directly controls how much parallelism Spark can exploit, so operations like repartition() and coalesce() let you tune partition count explicitly. When an RDD is reused across multiple actions, calling persist() or cache() stores the computed partitions in memory (or disk) so Spark avoids recomputing the entire lineage chain every single time.
Cricket analogy: It's like a captain deciding how many net bowlers to deploy before a big match; too few and batsmen face too little variety, too many and you waste practice-ground space, and once a good net session is filmed, the team rewatches the recording instead of rebowling live.
Calling .cache() or .persist() on every RDD you touch is a common beginner mistake. Cached partitions occupy executor memory, and if you cache more data than fits, Spark will evict older cached partitions or spill to disk, which can slow the job down instead of speeding it up. Only cache RDDs that are reused across multiple actions.
- An RDD is an immutable, partitioned collection distributed across executors; transformations always return a new RDD rather than mutating the original.
- RDDs are created either from an in-memory collection via sc.parallelize() or from external storage via sc.textFile() and similar loaders.
- Fault tolerance comes from lineage: Spark records the DAG of transformations and recomputes only lost partitions instead of replicating data.
- Partition count controls parallelism; repartition() and coalesce() let you tune it, with coalesce() being cheaper for reducing partitions.
- persist()/cache() store computed partitions to avoid recomputing lineage on repeated actions, but should be used selectively to avoid memory pressure.
- RDDs remain the low-level foundation of Spark, but the DataFrame/Dataset API is preferred for most application code because it adds Catalyst optimization.
Practice what you learned
1. How does Spark recover a lost RDD partition after an executor failure?
2. Which method creates an RDD directly from an in-memory Python list?
3. What is the main risk of calling .cache() on every RDD in a pipeline?
4. Why should most new Spark application code prefer the DataFrame/Dataset API over raw RDDs?
Was this page helpful?
You May Also Like
Transformations and Actions
How Spark's lazy execution model splits operations into transformations that build a plan and actions that trigger real computation.
DataFrames Basics
How Spark's DataFrame API represents distributed, schema-aware tabular data and why it outperforms hand-written RDD code.
Schemas and Data Types
How Spark defines DataFrame structure with StructType schemas and a rich type system, and why explicit schemas matter in production.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics