Cluster Managers: YARN and Kubernetes
Spark itself only defines how a distributed computation is split into tasks; it relies on a cluster manager to actually acquire CPU and memory across machines and launch executor processes. Spark supports a built-in standalone manager, Apache Mesos (now deprecated), YARN, and Kubernetes, with YARN and Kubernetes being the two dominant choices in production because they support multi-tenant resource sharing, queues or namespaces, and security integration.
Cricket analogy: Just as a franchise's team owner doesn't personally negotiate stadium bookings and ground staff but delegates it to the venue authority, Spark delegates the job of finding grounds (machines) and pitches (executors) to a cluster manager like the IPL's venue committee handling multiple matches.
Running Spark on YARN
On YARN, the Spark driver in cluster mode runs inside an ApplicationMaster container that YARN's ResourceManager launches on any NodeManager in the cluster, while in client mode the driver stays on the submitting machine and only the ApplicationMaster runs in the cluster. The ResourceManager's scheduler (typically the Capacity Scheduler or Fair Scheduler) grants containers on NodeManagers for Spark executors, and dynamic allocation relies on YARN's external shuffle service so executors can be removed without losing shuffle data that downstream stages still need.
Cricket analogy: In an IPL auction, the BCCI's auction committee (ResourceManager) decides which franchise gets which player slot, while a franchise's coach in the dugout (Spark driver in client mode) directs strategy without owning the auction process itself.
spark-submit \
--master yarn \
--deploy-mode cluster \
--num-executors 20 \
--executor-memory 8g \
--executor-cores 4 \
--queue production \
--class com.example.SalesETL \
s3://my-bucket/jobs/sales-etl.jar \
--input s3://my-bucket/raw/sales/ \
--output s3://my-bucket/curated/sales/Running Spark on Kubernetes
On Kubernetes, spark-submit talks directly to the Kubernetes API server, which schedules a driver pod; that driver pod then creates executor pods itself via the API rather than going through an intermediate resource negotiator like YARN's ApplicationMaster. Executors run as regular pods governed by Kubernetes namespaces, resource quotas, and RBAC service accounts, and pod templates let teams attach sidecars, node selectors, or tolerations so Spark workloads follow the same governance as other containerized services in the cluster.
Cricket analogy: A modern franchise using a data-driven front office lets the head coach directly request specific net bowlers from the academy pool without going through a separate board committee, similar to how the Spark driver pod requests executor pods directly from the Kubernetes API.
spark-submit \
--master k8s://https://k8s-apiserver.example.com:6443 \
--deploy-mode cluster \
--name sales-etl \
--conf spark.kubernetes.namespace=data-eng \
--conf spark.kubernetes.container.image=myregistry/spark:3.5.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
--conf spark.executor.instances=20 \
--class com.example.SalesETL \
local:///opt/spark/jobs/sales-etl.jarDynamic allocation on Kubernetes needs an external shuffle service equivalent — either Spark's Kubernetes shuffle tracking (spark.dynamicAllocation.shuffleTracking.enabled) or a dedicated shuffle service DaemonSet. Without it, scaling executors down can silently drop shuffle files that later stages still need, causing FetchFailedException errors mid-job.
Choosing the Right Cluster Manager
Choosing between YARN and Kubernetes usually comes down to existing infrastructure and operational model: organizations already running Hadoop with HDFS and Hive typically keep Spark on YARN to share the same queues, capacity scheduler, and data locality benefits, while cloud-native teams standardizing on Kubernetes for all workloads prefer running Spark there to get a single control plane, container-level isolation, and easier autoscaling via cluster autoscalers. YARN's Capacity Scheduler offers mature multi-tenant queue management with preemption, whereas Kubernetes offers stronger workload isolation through containers and namespaces but historically weaker native support for Spark's external shuffle service, which improved with Kubernetes shuffle tracking and dynamic allocation features in Spark 3.x.
Cricket analogy: A team built around traditional red-ball first-class talent naturally stays with domestic first-class cricket infrastructure, while a franchise built for T20 leagues invests in a completely different training and scouting setup, just as YARN suits Hadoop-native shops and Kubernetes suits container-native ones.
YARN's Capacity Scheduler organizes cluster resources into hierarchical queues (e.g., production, adhoc, ml) with guaranteed minimum capacity and optional elasticity to borrow idle capacity from other queues — a mature multi-tenancy model that Kubernetes ResourceQuotas and namespaces approximate but don't replicate exactly.
- Spark relies on an external cluster manager — standalone, YARN, or Kubernetes — to actually acquire machines and launch executors.
- On YARN, the ResourceManager grants containers on NodeManagers, and the Spark driver runs as (or alongside) a YARN ApplicationMaster.
- On Kubernetes, the driver runs as a pod and creates executor pods directly via the Kubernetes API, without an intermediate ApplicationMaster layer.
- Dynamic allocation requires an external shuffle mechanism on both YARN (shuffle service) and Kubernetes (shuffle tracking or a shuffle service DaemonSet).
- YARN suits organizations already invested in Hadoop/HDFS infrastructure with mature multi-tenant queue scheduling.
- Kubernetes suits cloud-native, containerized environments that want a single control plane and stronger workload isolation.
Practice what you learned
1. In YARN cluster deploy mode, where does the Spark driver run?
2. How does the Spark driver acquire executors when running on Kubernetes?
3. What is required for Spark's dynamic allocation to safely scale down executors without losing needed shuffle data?
4. Which factor most commonly drives an organization to choose YARN over Kubernetes for Spark?
Was this page helpful?
You May Also Like
spark-submit and Deployment
Master the spark-submit command-line tool and the client vs. cluster deployment modes for running Spark applications.
Monitoring with the Spark UI
Navigate the Spark web UI to diagnose job performance, stage bottlenecks, and executor behavior in running and completed applications.
Performance Tuning
Practical techniques for tuning Spark jobs: partitioning, caching, join strategies, memory configuration, and avoiding data skew.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics