Caching and Persistence

Analogy categoryAnalogies

Why Cache a DataFrame

When a DataFrame is reused across multiple actions, for example computing several different aggregations on the same filtered dataset, Spark recomputes the entire lineage from source each time unless you explicitly cache it. Calling .cache() or .persist() materializes the DataFrame's partitions in memory (or disk) after the first action touches them, so subsequent actions read the cached partitions directly instead of replaying transformations and re-reading from the source. This trades memory for compute time, and it only pays off when a DataFrame is actually accessed more than once.

🏏

Cricket analogy: It's like a commentator keeping a player's full career stats sheet open on the desk instead of recalculating batting average from scorecards every time it's mentioned — pulled up once, reused for the rest of the broadcast.

Storage Levels: Memory, Disk, and Serialization

persist() accepts a StorageLevel that controls where and how data is stored: MEMORY_ONLY keeps deserialized Java objects in RAM for the fastest access but risks partitions being dropped if memory runs out, MEMORY_AND_DISK spills evicted partitions to local disk instead of losing them, and adding _SER suffixes stores data in a compact serialized byte format that saves memory at the cost of CPU time to deserialize on read. Choosing the right level is a tradeoff between speed, memory pressure, and how expensive recomputation would be if a partition were lost.

🏏

Cricket analogy: It's like choosing between keeping your full kit bag in the dressing room versus a locker down the hall — the dressing room is instant access but limited space, the locker holds more but costs a walk every time you need gear.

python

from pyspark.sql import StorageLevel

customers = spark.read.parquet("s3://data/customers.parquet")
active = customers.filter(customers.status == "active")

# Cache in memory, spilling to disk if it doesn't fit
active.persist(StorageLevel.MEMORY_AND_DISK)

# First action materializes the cache
active.count()

# Subsequent actions reuse the cached partitions
by_region = active.groupBy("region").count()
by_plan = active.groupBy("plan_tier").count()

# Release the cache once you're done reusing it
active.unpersist()

When Caching Backfires

Caching every DataFrame indiscriminately is a common anti-pattern: memory is finite, and caching a large intermediate result that's only used once provides zero benefit while evicting other, genuinely reused cached data under Spark's least-recently-used eviction policy. Over-caching also increases garbage collection pressure on the JVM heap, since cached objects live longer and compete with the execution memory Spark needs for shuffles and joins, sometimes causing OutOfMemory errors or forcing expensive spills to disk that negate the entire point of caching.

🏏

Cricket analogy: It's like a team management insisting on flying in every reserve player for every single match regardless of whether they'll play — the squad hotel fills up, and it crowds out room for the players who actually need to be there.

Spark does not automatically detect that a cached DataFrame has gone stale when the underlying source changes or you rebuild it with different transformations — the cache key is the logical plan, not a live link to the data. Call unpersist() explicitly whenever you're done reusing a cached DataFrame or its lineage changes.

Unpersisting and the Cache Lifecycle

A cached DataFrame remains in memory until you call unpersist(), the Spark application ends, or the block manager evicts it under memory pressure — Spark does not automatically garbage-collect caches just because you stop referencing the variable in your code. If the underlying source data changes, or you rebuild the DataFrame with a different transformation chain, the old cached version becomes stale and should be unpersisted explicitly rather than relying on Spark to detect the change, since Spark's cache key is based on the logical plan, not a live connection to the source.

🏏

Cricket analogy: It's like a scoreboard that keeps displaying yesterday's final total until someone manually resets it — the ground staff have to explicitly clear it before the new match, it doesn't wipe itself just because play ended.

The Spark UI's Storage tab (under the application's web UI) lists every cached RDD/DataFrame along with its storage level, the fraction actually cached in memory versus spilled to disk, and total size — use it to confirm a cache is fully materialized rather than partially evicted.

cache()/persist() materialize a DataFrame after its first action so later actions skip recomputation.
Caching only helps when a DataFrame is reused across more than one action.
MEMORY_AND_DISK spills evicted partitions to disk instead of dropping them; _SER levels trade CPU for memory.
Over-caching increases GC pressure and can evict genuinely reused data under LRU eviction.
Cached data persists until unpersist() is called, the app ends, or it's evicted under memory pressure.
The Spark UI Storage tab shows exactly what's cached and how much memory/disk it's using.