Apache Spark (PySpark) Cheat Sheet
PySpark reference covering SparkSession setup, DataFrame transformations and actions, Spark SQL queries, caching, and writing partitioned output.
2 PagesIntermediateMar 30, 2026
SparkSession & DataFrame
Read data and run basic operations.
python
from pyspark.sql import SparkSessionspark = SparkSession.builder.appName("MyApp").getOrCreate()df = spark.read.csv("data.csv", header=True, inferSchema=True)df.printSchema()df.show(5)df.select("name", "age").filter(df.age > 21).show()
Transformations
GroupBy, joins, and column expressions.
python
from pyspark.sql import functions as Fresult = ( df.groupBy("department") .agg(F.avg("salary").alias("avg_salary"), F.count("*").alias("count")) .orderBy(F.desc("avg_salary")))df2 = df.withColumn("bonus", F.col("salary") * 0.1)df3 = df.dropna(subset=["age"])joined = df.join(other_df, on="employee_id", how="left")
Spark SQL & Caching
Query with SQL and persist intermediate results.
python
df.createOrReplaceTempView("employees")spark.sql("SELECT department, AVG(salary) FROM employees GROUP BY department").show()df.cache() # persist in memory across actionsdf.repartition(8) # increase parallelismdf.write.mode("overwrite").parquet("output/")
Core Concepts
Fundamental Spark building blocks.
- SparkSession- unified entry point for the DataFrame/SQL API
- DataFrame- distributed, immutable table of structured data
- RDD- low-level resilient distributed dataset (rarely used directly today)
- transformations- lazy ops like select/filter/groupBy that build a query plan
- actions- trigger execution, e.g. show()/collect()/count()
- partition- unit of parallelism distributed across the cluster
- Catalyst optimizer- Spark SQL's query optimization engine
Window Functions
Rank, rolling-aggregate, and compare rows within partitions.
python
from pyspark.sql import Windowfrom pyspark.sql import functions as Fwindow_spec = Window.partitionBy("department").orderBy(F.desc("salary"))ranked = df.withColumn("rank", F.rank().over(window_spec)) \ .withColumn("running_total", F.sum("salary").over(window_spec.rowsBetween(Window.unboundedPreceding, Window.currentRow)))top_earners = ranked.filter(F.col("rank") <= 3)
Pandas UDFs (Vectorized)
Write Arrow-backed UDFs that operate on whole columns instead of row by row.
python
import pandas as pdfrom pyspark.sql.functions import pandas_udffrom pyspark.sql.types import DoubleType@pandas_udf(DoubleType())def celsius_to_fahrenheit(series: pd.Series) -> pd.Series: return series * 9 / 5 + 32 # vectorized via Arrow — far faster than a row-at-a-time UDFdf.withColumn("temp_f", celsius_to_fahrenheit(df.temp_c)).show()# plain (non-vectorized) UDF for comparison — avoid for large datafrom pyspark.sql.functions import udffrom pyspark.sql.types import StringTypeslow_udf = udf(lambda x: x.upper(), StringType())
Broadcast Joins & Partitioning
Avoid costly shuffles by broadcasting small tables and tuning partition counts.
python
from pyspark.sql.functions import broadcast# force a broadcast join when one side is small (avoids a costly shuffle)result = large_df.join(broadcast(small_df), on="id", how="inner")df.repartition(200, "department") # hash-repartition by key before a heavy groupBy/joindf.coalesce(1).write.csv("single_output/") # reduce partitions without a full shuffle (for output)print(df.rdd.getNumPartitions())
Structured Streaming
Process an unbounded data stream with the same DataFrame API as batch jobs.
python
stream_df = ( spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "events") .load())parsed = stream_df.selectExpr("CAST(value AS STRING) as json") \ .select(F.from_json("json", schema).alias("data")).select("data.*")query = ( parsed.groupBy(F.window("event_time", "5 minutes"), "category") .count() .writeStream .outputMode("update") .format("console") .trigger(processingTime="1 minute") .start())query.awaitTermination()
Performance Tuning Knobs
Configuration options that most affect Spark job performance.
- spark.sql.adaptive.enabled (AQE)- dynamically coalesces shuffle partitions and re-optimizes joins at runtime
- spark.sql.shuffle.partitions- controls shuffle output partition count (default 200); tune down for small data, up for large
- spark.sql.autoBroadcastJoinThreshold- tables under this size are auto-broadcast instead of shuffled during a join
- AQE skew join handling- automatically splits oversized shuffle partitions so one skewed key doesn't stall a stage
- persist(StorageLevel...)- like cache() but lets you choose memory-only, disk-only, or serialized storage levels
- explain(mode="formatted")- inspects the physical/logical plan Catalyst generates to spot unnecessary shuffles and scans
- checkpoint()- truncates lineage for long iterative jobs to avoid huge DAGs and stack overflows
Pro Tip
DataFrame transformations are lazy — nothing executes until an action like show(), collect(), or write() is called, so chain multiple filters/selects freely without worrying about intermediate computation cost.
Was this cheat sheet helpful?
Explore Topics
#ApacheSparkPySpark#ApacheSparkPySparkCheatSheet#DataScience#Intermediate#SparkSessionDataFrame#Transformations#SparkSQLCaching#CoreConcepts#Databases#MachineLearning#CheatSheet#SkillVeris
