Your First Spark Job

Analogy categoryAnalogies

A typical Spark job follows a read -> transform -> action pattern: you load data into a DataFrame or RDD, chain a series of transformations like filter and groupBy, and finish with an action that actually triggers execution. Crucially, transformations are lazy - none of them process a single row of data when called; they simply build up a logical plan.

🏏

Cricket analogy: A batting innings only truly registers on the scoreboard once a run is completed between wickets, not when the bat merely makes contact, mirroring how Spark transformations like filter or map are lazily queued and only execute when an action like collect() triggers them.

Transformations vs Actions

Transformations split into narrow ones, like map() and filter(), which operate entirely within a single partition, and wide ones, like groupBy() and join(), which require a shuffle to bring matching keys together across partitions. Actions like show(), count(), collect(), and write() are what actually trigger the DAG scheduler to execute the plan across the cluster.

🏏

Cricket analogy: A fielder repositioning within their own zone (narrow transformation like map or filter) needs no coordination with others, but a full field change requiring every player to swap positions (wide transformation like groupByKey) needs the whole team to communicate via the captain.

A Complete Read-Transform-Write Job

A realistic first Spark job reads a CSV of orders into a DataFrame, filters it down to one category, groups and aggregates by region, and finally writes the result out to Parquet. Only when write() (an action) is called does Spark's Catalyst optimizer finalize a physical plan and the DAG scheduler dispatch tasks to executors.

🏏

Cricket analogy: Compiling a season's full scorecards from every match (reading data), filtering to just one team's innings (transform), and printing the final batting average (action/output) mirrors a typical Spark job reading a CSV, filtering rows, and writing aggregated results.

python

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg

spark = SparkSession.builder.appName("FirstJob").master("local[*]").getOrCreate()

orders = spark.read.option("header", True).option("inferSchema", True).csv("data/orders.csv")

# Transformations (lazy - nothing runs yet)
electronics = orders.filter(col("category") == "Electronics")
summary = electronics.groupBy("region").agg(avg("amount").alias("avg_order_value"))

# Action - this triggers the actual DAG execution
summary.orderBy(col("avg_order_value").desc()).show()

summary.write.mode("overwrite").parquet("output/electronics_by_region")

spark.stop()

filter(), groupBy(), and agg() are all transformations - none of them touch a single row of data when called. Spark just builds up a logical plan. Nothing actually executes until an action like show(), count(), collect(), or write() is called, at which point Spark's Catalyst optimizer finalizes the physical plan and the DAG runs across executors.

A Spark job typically follows a read -> transform -> action pattern.
Transformations (filter, map, groupBy) are lazy and only build a logical execution plan.
Actions (show, count, collect, write) trigger actual computation across the cluster.
Narrow transformations (map, filter) need no shuffle; wide transformations (groupBy, join) do.
Lazy evaluation lets Spark's Catalyst optimizer rewrite the whole plan before executing anything.
DataFrames read with inferSchema or an explicit schema, and can be written to Parquet, CSV, or JSON.

Practice what you learned

1. In Spark, when does a transformation like filter() actually process data?

2. Which of these is an action, not a transformation?

3. What distinguishes a wide transformation from a narrow one?

4. What is the benefit of Spark's lazy evaluation model?

5. Which method call actually triggers a Spark job to run in this snippet: df.filter(...).groupBy(...).count()?

Was this page helpful?

The SparkSession

Related Study Notes in Programming

Browse all study notes

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheSparkStudyNotes#YourFirstSparkJob#Spark#Job#Transformations#Actions#StudyNotes#SkillVeris#ExamPrep

Your First Spark Job