Spark with PySpark

PySpark lets you write Spark applications in Python, but under the hood, DataFrame and SQL operations are translated into the same JVM logical and physical execution plans that Scala or Java Spark code produces — the Python API is essentially a client that builds a query plan and hands it to the JVM driver, so calling df.filter(...).groupBy(...).agg(...) incurs no per-row Python overhead because the actual execution happens entirely in JVM executors. This is different from row-at-a-time Python UDFs, which do require shipping data out to a separate Python process.

PySpark's driver process talks to the JVM driver through Py4J, a bridge that lets Python objects invoke Java methods, but this bridge becomes a real cost the moment you register a plain Python UDF with spark.sql.functions.udf: for every row, Spark must serialize the row's data out of the JVM, pass it to a separate Python worker process, execute your function, and serialize the result back, which is far slower than JVM-native execution. Pandas UDFs (spark.sql.functions.pandas_udf), built on Apache Arrow, dramatically cut this overhead by batching many rows into a columnar Arrow buffer and handing the whole batch to Python at once via vectorized pandas operations instead of one row at a time.

The fastest PySpark code is usually the code that uses the least Python: prefer built-in functions from pyspark.sql.functions (col, when, regexp_extract, date_trunc, and dozens more) over UDFs whenever an equivalent exists, because these compile into the JVM's Catalyst-optimized expressions and can be pushed down or reordered by the optimizer, unlike opaque UDF logic. Chaining select() and withColumn() calls to build up a transformation is fine — Spark's Catalyst optimizer combines them into an efficient physical plan regardless of how many intermediate DataFrame variables you create — but calling .collect() or .toPandas() on anything beyond a small, already-aggregated result pulls the entire dataset into the driver's single JVM process and risks an out-of-memory crash.

Because executors run Python worker processes, the Python interpreter and package versions matter on every node, not just the driver: spark.pyspark.python (and spark.pyspark.driver.python) tell Spark which Python binary to launch, and mismatched versions between driver and executors are a common source of confusing pickling errors. For cluster deployments where you can't guarantee every node has the right packages pre-installed, tools like venv-pack or conda-pack let you bundle an entire Python virtual environment into a single archive that spark-submit ships to every executor via --archives, guaranteeing identical dependencies everywhere without manually provisioning each machine.