Performance Tuning

Analogy categoryAnalogies

Most Spark performance problems trace back to a handful of root causes: too much or too little shuffle partitioning, data skew that leaves some tasks doing far more work than others, spilling intermediate data to disk because memory is undersized, or reading too many small files that create excessive task-scheduling overhead. Effective tuning starts with reading the Spark UI to identify which of these is actually happening rather than guessing and randomly changing configuration values.

🏏

Cricket analogy: A team's poor performance in a series is usually traceable to one or two specific root causes — a weak middle order or poor death bowling — rather than everything being wrong at once, just as Spark slowness usually traces to a specific bottleneck like skew or spill.

Partitioning and Shuffle Tuning

The number of shuffle partitions, controlled by spark.sql.shuffle.partitions (default 200), directly determines task parallelism after any shuffle: too few partitions on a large dataset creates huge tasks that spill to disk, while too many on a small dataset creates thousands of tiny tasks dominated by scheduling overhead. Adaptive Query Execution (AQE), enabled by default since Spark 3.x via spark.sql.adaptive.enabled, mitigates this by coalescing small shuffle partitions at runtime based on actual data size rather than requiring a hand-tuned static value, though explicit repartition(n) or repartition(n, col) calls are still useful to control partitioning before writes or to fix a specific skewed stage.

🏏

Cricket analogy: Fielding with only 3 players spread across a huge boundary leaves gaps everywhere, while overcrowding with 20 players in a tiny circle is equally useless — the right number of shuffle partitions is like setting the right number of fielders for the ground size.

python

spark = (SparkSession.builder
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .getOrCreate())

# Explicit repartition before a large write to control output file count
df = df.repartition(200, "customer_region")
df.write.mode("overwrite").parquet("s3://bucket/curated/orders/")

Caching and Memory Management

Spark's unified memory manager splits executor JVM heap into a storage region (for cached RDDs/DataFrames) and an execution region (for shuffles, joins, and aggregations), with the boundary able to flex based on demand — persist() or cache() with storage levels like MEMORY_AND_DISK write cached partitions to disk once memory is full rather than dropping them, at the cost of slower reads on eviction. Choosing MEMORY_ONLY for data that fits comfortably avoids serialization overhead, while MEMORY_AND_DISK_SER trades some CPU for a smaller memory footprint when the cached dataset is large relative to available heap.

🏏

Cricket analogy: A team keeping its best all-rounders on standby in the dugout for quick access during a chase, but sending lesser-used reserves to the pavilion when space is tight, mirrors how Spark evicts less-used cached partitions to disk under memory pressure.

Caching every intermediate DataFrame 'just in case' is a common anti-pattern: since storage and execution memory share the same unified region, aggressively caching large datasets can evict data other stages need or push execution into disk spill, making the job slower overall rather than faster. Only cache DataFrames that are reused multiple times downstream.

Join Strategies and Skew Handling

Spark automatically uses a broadcast hash join when one side of a join is smaller than spark.sql.autoBroadcastJoinThreshold (10MB by default), copying the small table to every executor and avoiding a shuffle entirely, which is dramatically faster than the default sort-merge join used for two large tables. When a sort-merge join suffers from skew — one join key value dominating the dataset — Adaptive Query Execution's skew join optimization automatically splits the oversized partition into smaller sub-partitions at runtime, and for cases AQE doesn't catch, manually salting the skewed key (appending a random suffix to spread it across more partitions) remains a reliable technique.

🏏

Cricket analogy: Sending just the specialist death-bowling coach to every net session instead of moving the entire squad's training setup each time is like broadcasting a small table to every executor instead of shuffling the whole large dataset.

Adaptive Query Execution (AQE) has been enabled by default since Spark 3.2 (spark.sql.adaptive.enabled=true). It re-optimizes the query plan at runtime using actual shuffle statistics, handling partition coalescing and skew join splitting automatically in most cases — manual tuning like static shuffle.partitions values or salting is now mainly needed for edge cases AQE doesn't catch.

Most Spark performance problems trace to shuffle sizing, data skew, memory spill, or too many small files.
spark.sql.shuffle.partitions controls post-shuffle task parallelism; both too few and too many partitions hurt performance.
Adaptive Query Execution (AQE), on by default since Spark 3.2, dynamically coalesces shuffle partitions and splits skewed ones at runtime.
Spark's unified memory manager shares heap between storage (caching) and execution (shuffles/joins), flexing the boundary based on demand.
MEMORY_AND_DISK spills cached partitions to disk once memory is full instead of dropping them, trading speed for durability.
Broadcast joins avoid shuffles entirely for tables under spark.sql.autoBroadcastJoinThreshold (10MB default).
Manual key salting remains useful for skew cases AQE's automatic skew join optimization doesn't fully resolve.