Why Cache a DataFrame
When a DataFrame is reused across multiple actions, for example computing several different aggregations on the same filtered dataset, Spark recomputes the entire lineage from source each time unless you explicitly cache it. Calling .cache() or .persist() materializes the DataFrame's partitions in memory (or disk) after the first action touches them, so subsequent actions read the cached partitions directly instead of replaying transformations and re-reading from the source. This trades memory for compute time, and it only pays off when a DataFrame is actually accessed more than once.
Cricket analogy: It's like a commentator keeping a player's full career stats sheet open on the desk instead of recalculating batting average from scorecards every time it's mentioned — pulled up once, reused for the rest of the broadcast.
Storage Levels: Memory, Disk, and Serialization
persist() accepts a StorageLevel that controls where and how data is stored: MEMORY_ONLY keeps deserialized Java objects in RAM for the fastest access but risks partitions being dropped if memory runs out, MEMORY_AND_DISK spills evicted partitions to local disk instead of losing them, and adding _SER suffixes stores data in a compact serialized byte format that saves memory at the cost of CPU time to deserialize on read. Choosing the right level is a tradeoff between speed, memory pressure, and how expensive recomputation would be if a partition were lost.
Cricket analogy: It's like choosing between keeping your full kit bag in the dressing room versus a locker down the hall — the dressing room is instant access but limited space, the locker holds more but costs a walk every time you need gear.
from pyspark.sql import StorageLevel
customers = spark.read.parquet("s3://data/customers.parquet")
active = customers.filter(customers.status == "active")
# Cache in memory, spilling to disk if it doesn't fit
active.persist(StorageLevel.MEMORY_AND_DISK)
# First action materializes the cache
active.count()
# Subsequent actions reuse the cached partitions
by_region = active.groupBy("region").count()
by_plan = active.groupBy("plan_tier").count()
# Release the cache once you're done reusing it
active.unpersist()
When Caching Backfires
Caching every DataFrame indiscriminately is a common anti-pattern: memory is finite, and caching a large intermediate result that's only used once provides zero benefit while evicting other, genuinely reused cached data under Spark's least-recently-used eviction policy. Over-caching also increases garbage collection pressure on the JVM heap, since cached objects live longer and compete with the execution memory Spark needs for shuffles and joins, sometimes causing OutOfMemory errors or forcing expensive spills to disk that negate the entire point of caching.
Cricket analogy: It's like a team management insisting on flying in every reserve player for every single match regardless of whether they'll play — the squad hotel fills up, and it crowds out room for the players who actually need to be there.
Spark does not automatically detect that a cached DataFrame has gone stale when the underlying source changes or you rebuild it with different transformations — the cache key is the logical plan, not a live link to the data. Call unpersist() explicitly whenever you're done reusing a cached DataFrame or its lineage changes.
Unpersisting and the Cache Lifecycle
A cached DataFrame remains in memory until you call unpersist(), the Spark application ends, or the block manager evicts it under memory pressure — Spark does not automatically garbage-collect caches just because you stop referencing the variable in your code. If the underlying source data changes, or you rebuild the DataFrame with a different transformation chain, the old cached version becomes stale and should be unpersisted explicitly rather than relying on Spark to detect the change, since Spark's cache key is based on the logical plan, not a live connection to the source.
Cricket analogy: It's like a scoreboard that keeps displaying yesterday's final total until someone manually resets it — the ground staff have to explicitly clear it before the new match, it doesn't wipe itself just because play ended.
The Spark UI's Storage tab (under the application's web UI) lists every cached RDD/DataFrame along with its storage level, the fraction actually cached in memory versus spilled to disk, and total size — use it to confirm a cache is fully materialized rather than partially evicted.
- cache()/persist() materialize a DataFrame after its first action so later actions skip recomputation.
- Caching only helps when a DataFrame is reused across more than one action.
- MEMORY_AND_DISK spills evicted partitions to disk instead of dropping them; _SER levels trade CPU for memory.
- Over-caching increases GC pressure and can evict genuinely reused data under LRU eviction.
- Cached data persists until unpersist() is called, the app ends, or it's evicted under memory pressure.
- The Spark UI Storage tab shows exactly what's cached and how much memory/disk it's using.
Practice what you learned
1. What does calling .cache() on a DataFrame actually do?
2. When is caching a DataFrame actually beneficial?
3. What does the MEMORY_AND_DISK storage level do that MEMORY_ONLY does not?
4. What is a risk of caching too many DataFrames indiscriminately?
5. Why must you call unpersist() explicitly rather than relying on Spark to detect a cache is stale?
Was this page helpful?
You May Also Like
Lazy Evaluation and the DAG
How Spark defers execution of transformations into a logical plan and only runs work when an action forces it, using a DAG of stages for both optimization and fault tolerance.
Partitioning and Shuffling
How Spark distributes data into partitions for parallelism, why shuffles are the costliest operation in a job, and how to control partition count and skew.
Joins in Spark
How Catalyst chooses between broadcast hash joins and sort-merge joins, why the choice matters for performance, and how to diagnose and fix skewed joins.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics