Spark Architecture

Analogy categoryAnalogies

Every Spark application has one driver process and a set of executor processes distributed across a cluster, coordinated with the help of a cluster manager (standalone, YARN, Kubernetes, or Mesos). The driver decides what work needs to happen and where; the cluster manager decides which physical machines get to run that work; the executors actually do it.

🏏

Cricket analogy: Spark's architecture works like a cricket captain such as Rohit Sharma (the driver) planning field placements while the team management office (cluster manager like YARN) assigns which players (executors) stand at which position on the ground.

The Driver and SparkContext

The driver runs the application's main() function and hosts the SparkContext (wrapped today by SparkSession), which is responsible for converting the user's chain of transformations into a logical DAG of stages before any executor does any work. The driver also tracks the state of every task and re-schedules failed ones.

🏏

Cricket analogy: The team analyst preparing a detailed over-by-over plan before the match, the way Spark's driver process builds a DAG of transformations via SparkContext before any executor starts working, resembles pre-match strategy sessions run by coaches like Gautam Gambhir.

Executors, Tasks, and Partitions

Executors are JVM processes launched on worker nodes that run the actual tasks the driver schedules and cache data in memory or disk for the application's lifetime. Each dataset is split into partitions, and each task processes exactly one partition, which is how Spark achieves parallelism across many CPU cores at once.

🏏

Cricket analogy: Each fielder covering their assigned zone of the outfield independently while the ball is in play resembles Spark executors each processing their own data partition as a task in parallel.

bash

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-cores 4 \
  --executor-memory 8g \
  --class com.example.SalesETL \
  sales-etl-1.0.jar

The driver is a single point of coordination - if it crashes, the whole application fails, even though executors are distributed. In cluster deploy mode, the driver itself runs inside the cluster (not on your laptop) so a client disconnect won't kill the job, but the driver process itself is still not fault-tolerant the way executors are.

Spark applications consist of one driver process and many executor processes.
The driver runs the user's main() function, builds the DAG, and schedules tasks.
A cluster manager (standalone, YARN, Kubernetes, or Mesos) allocates resources for executors.
Executors run tasks in parallel, each processing one partition of data at a time.
The driver is a single point of failure for the whole Spark application.
Wide transformations trigger shuffles that move data across executors over the network.

Practice what you learned

1. What is the role of the Spark driver?

2. What does an executor do in Spark?

3. Which of these is NOT a valid Spark cluster manager?

4. In spark-submit, what does --deploy-mode cluster do?

Was this page helpful?

What Is Apache Spark?

Installing and Running Spark

Related Study Notes in Programming

Browse all study notes

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheSparkStudyNotes#SparkArchitecture#Spark#Architecture#Driver#SparkContext#StudyNotes#SkillVeris#ExamPrep

Spark Architecture