Partitioning and shuffling are the two most performance-critical internal mechanics of Spark. Partitioning determines how data is distributed across executor cores — the degree of parallelism that makes distributed computation fast. Shuffling is the expensive redistribution of data required by wide transformations, and it is the dominant performance bottleneck in most production Spark jobs. Data skew — where a small number of partitions contain a disproportionately large fraction of the data — turns a well-parallelised job into a sequential bottleneck.
Understanding partitioning, shuffling, and skew enables a Spark engineer to diagnose the most common production performance problem: a job that uses 90% of executor cores for most of its duration but takes three times longer than expected because the final reduce stage has one task running for 20 minutes while 199 others finished in 30 seconds. Solving this requires recognising the skew, identifying its source key, and applying the correct mitigation rather than adding more hardware to the cluster.