User-Defined Functions (UDFs) and Pandas UDFs — Big Data & Distributed Computing

Built-in Spark functions cover the vast majority of column transformation needs, but occasionally a business rule cannot be expressed with built-ins and requires custom Python logic. User-Defined Functions (UDFs) let Python functions be applied to DataFrame columns, but they cross from the JVM's Tungsten execution engine into Python's runtime, serialising each row through pickle — a process that can be 10 to 100 times slower than equivalent built-in functions. Understanding when to use UDFs and when Pandas UDFs provide a faster alternative is essential for any PySpark practitioner.

Pandas UDFs solve the row-at-a-time serialisation problem by operating on pandas Series or DataFrames — batches of rows — rather than individual values. Apache Arrow transfers data between the JVM and Python in a zero-copy columnar format, eliminating pickle serialisation entirely. The result is a UDF that is 10 to 100 times faster than a standard Python UDF while still allowing arbitrary Python and pandas logic. Pandas UDFs are the correct choice for any custom transformation logic that must operate on batches of rows efficiently.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.