Monitoring with the Spark UI

Every running Spark application exposes a web UI, by default on port 4040 on the driver (incrementing to 4041, 4042, etc. if multiple applications run on the same machine), with tabs covering Jobs, Stages, Storage, Environment, Executors, and — for DataFrame/SQL workloads — a dedicated SQL tab showing physical query plans. This UI is the primary tool for understanding why a job is slow, whether it's failing due to memory pressure, or which stage is the bottleneck, and it's available for the lifetime of the driver process.

The Jobs tab lists each Spark action (like collect, count, or save) as a job broken into stages at shuffle boundaries, and clicking into a stage reveals a task-level table plus summary metrics — task duration distribution, shuffle read/write size, and spill to disk — that is the fastest way to spot skew: if the max task duration is far larger than the median, a handful of partitions are doing disproportionate work, usually from skewed join or group-by keys. The DAG visualization for each job shows how RDDs or DataFrame operations chain together across stages, making it easy to see exactly where a wide transformation forces a shuffle.

The Executors tab shows, per executor, active/completed task counts, total GC time, storage memory used for cached data, and shuffle read/write, which is essential for spotting garbage-collection pressure (high GC time relative to task time usually means executor memory is too small or too many objects are being cached) or an executor that died and was replaced. Because the default UI on port 4040 only exists while the driver process is alive, production clusters set spark.eventLog.enabled=true and spark.eventLog.dir to a persistent location like HDFS or S3, then run the standalone Spark History Server so completed applications' UIs remain browsable long after the job finishes.

For DataFrame and Spark SQL workloads, the SQL tab visualizes the physical execution plan as a graph of operators like Scan, Filter, Exchange (a shuffle), and SortMergeJoin or BroadcastHashJoin, with live row counts and data size annotated at each node — this is the clearest way to confirm whether Spark chose a broadcast join versus an expensive shuffle join, and to see exactly which Exchange node is moving the most data. Combined with the Stages tab, you can trace a slow SQL query down to the specific shuffle stage and skewed partition causing it, which is usually far faster than guessing from execution time alone.