Spark Interview Questions

Analogy categoryAnalogies

Core Concepts Interviewers Probe

Interviewers consistently probe whether a candidate understands Spark's core abstractions: RDDs are the low-level, type-unsafe distributed collection with no built-in optimization, DataFrames add a schema and let Catalyst optimize the query plan, and Datasets (JVM languages only) combine DataFrame optimization with compile-time type safety. Equally fundamental is the transformation-versus-action distinction - transformations like map or filter are lazily recorded into a DAG and don't execute until an action like collect, count, or write triggers the actual computation.

🏏

Cricket analogy: Explaining the difference between a raw scorecard (RDD), a structured analytics dashboard (DataFrame), and a type-checked scoring app (Dataset) tests whether a candidate grasps Spark's abstraction layers, just as an interviewer probes RDD versus DataFrame versus Dataset.

Questions on Performance Tuning

A strong answer to 'explain a shuffle' distinguishes narrow transformations, where each output partition depends on only one input partition (like map or filter), from wide transformations like groupByKey or a sort-merge join, where data must be redistributed across the network by key. Candidates are also expected to explain when to use a broadcast join - sending a small table's full copy to every executor to avoid shuffling the large table entirely - and to recognize this only works when the small side fits comfortably within spark.sql.autoBroadcastJoinThreshold.

🏏

Cricket analogy: A fielder catching a ball hit directly to their position (narrow transformation) reacts instantly, while a run-out requiring relay throws across the field to redistribute the ball (wide transformation/shuffle) takes much longer, as an interview answer might explain.

python

# Interview-style answer, demonstrated in code

# Narrow transformation: no shuffle, each output partition depends on one input partition
filtered = orders_df.filter(orders_df.status == "COMPLETED")

# Wide transformation: triggers a shuffle to group matching keys together
by_customer = orders_df.groupBy("customer_id").sum("amount")

# Broadcast join: avoids shuffling the large `orders_df` entirely
from pyspark.sql.functions import broadcast
small_countries = spark.read.parquet("s3://dim/countries/")  # a few hundred rows
enriched = orders_df.join(broadcast(small_countries), "country_code")

print(enriched.explain())  # look for BroadcastHashJoin in the physical plan

Questions on Fault Tolerance and Architecture

Candidates should be able to draw the driver-executor architecture from memory: the driver runs the user's main program, builds the DAG, and schedules tasks, while executors, launched by a cluster manager (YARN, Kubernetes, or Spark's own standalone manager), actually run tasks and cache data. For long-running streaming or iterative jobs with deep lineage chains, explaining checkpointing - periodically writing the RDD's actual data (not just its lineage) to reliable storage like HDFS or S3 - shows awareness of when lineage-based recovery alone becomes too slow to be practical.

🏏

Cricket analogy: A captain (driver) sets the game plan and field placements while players (executors) actually execute the deliveries and catches on the ground, coordinated through the match officials (cluster manager), a structure worth sketching out in an interview answer.

A strong interview answer always ties the concept back to a concrete failure scenario: 'if an executor dies mid-shuffle, the driver reschedules its tasks on another executor, and lost partitions are recomputed via lineage' shows deeper understanding than a definition alone.

Scenario-Based / System Design Questions

A common system-design prompt asks the candidate to design a pipeline joining a billion-row events table against a skewed dimension table where one key (say, a default 'unknown user' ID) accounts for 40% of rows; a strong answer proposes isolating that hot key, salting it into several sub-keys to spread it across partitions, and handling the rest of the join normally. Candidates should also be ready to explain Spark's unified memory management, where execution memory (for shuffles and joins) and storage memory (for caching) share a single region and can borrow from each other under configurable thresholds, rather than being statically partitioned.

🏏

Cricket analogy: Designing a fielding plan around one power-hitter who accounts for 40% of a team's runs, by stationing multiple fielders specifically around that one batter's zones while the rest of the field stays standard, mirrors salting a skewed join key.

A common mistake in interviews: confidently using 'repartition' and 'coalesce' interchangeably. repartition() does a full shuffle and can increase or decrease partitions; coalesce() avoids a shuffle but can only decrease partitions and may create uneven sizes.

Know RDD vs DataFrame vs Dataset, and the transformation-vs-action distinction, cold.
Explain narrow vs wide transformations and when a broadcast join avoids a costly shuffle.
Be able to sketch the driver-executor architecture and name the supported cluster managers.
Understand checkpointing as the fallback when lineage-based recovery becomes impractically deep.
Practice a skewed-key system-design answer: isolate, salt, and rejoin the hot key separately.
Understand Spark's unified memory management, where execution and storage memory share a region.
Don't confuse repartition() (full shuffle, can grow or shrink partitions) with coalesce() (no shuffle, shrink only).