Apache Spark Study Notes

Building an ETL Pipeline with Spark

A practical guide to designing extract, transform, and load stages in Spark, from schema-safe ingestion to atomic loads.

Caching and Persistence

When and how to cache a Spark DataFrame to avoid recomputation, the tradeoffs between storage levels, and the pitfalls of over-caching or forgetting to unpersi…

Cluster Managers: YARN and Kubernetes

Learn how Apache Spark integrates with YARN and Kubernetes to acquire and manage cluster resources for distributed jobs.

DataFrames Basics

How Spark's DataFrame API represents distributed, schema-aware tabular data and why it outperforms hand-written RDD code.

Installing and Running Spark

How to install Apache Spark's prerequisites, run it locally for development, and choose the right cluster mode for production.

Joins in Spark

How Catalyst chooses between broadcast hash joins and sort-merge joins, why the choice matters for performance, and how to diagnose and fix skewed joins.

Lazy Evaluation and the DAG

How Spark defers execution of transformations into a logical plan and only runs work when an action forces it, using a DAG of stages for both optimization and…

Monitoring with the Spark UI

Navigate the Spark web UI to diagnose job performance, stage bottlenecks, and executor behavior in running and completed applications.

Partitioning and Shuffling

How Spark distributes data into partitions for parallelism, why shuffles are the costliest operation in a job, and how to control partition count and skew.

Performance Tuning

Practical techniques for tuning Spark jobs: partitioning, caching, join strategies, memory configuration, and avoiding data skew.

Reading and Writing Data Sources

Learn how Spark's DataFrameReader and DataFrameWriter APIs handle formats like Parquet, CSV, and JDBC, plus schema handling and partitioned writes.

Resilient Distributed Datasets (RDDs)

The foundational Spark abstraction: an immutable, partitioned, fault-tolerant collection of objects distributed across a cluster.

Running SQL Queries on DataFrames

Learn how to register DataFrames as temporary views and query them with standard SQL syntax, and how SQL and the DataFrame API interoperate under the hood.

Schemas and Data Types

How Spark defines DataFrame structure with StructType schemas and a rich type system, and why explicit schemas matter in production.

Spark Architecture

How Spark's driver, cluster manager, and executors work together to run a distributed job, from DAG construction to parallel task execution.

Spark Best Practices

Practical guidelines for writing efficient, reliable, and maintainable Apache Spark jobs in production.

Spark Interview Questions

The core concepts, performance-tuning questions, and system-design scenarios that come up most often in Apache Spark interviews.

Spark Quick Reference

A condensed cheat sheet of Spark's core APIs, common transformations and actions, I/O options, and key configuration settings.

spark-submit and Deployment

Master the spark-submit command-line tool and the client vs. cluster deployment modes for running Spark applications.

Spark vs Hadoop MapReduce

How Spark's in-memory DAG execution model compares to classic Hadoop MapReduce in performance, fault tolerance, and use cases.

Spark with PySpark

Understand how PySpark bridges Python code to the JVM-based Spark engine, and how to write efficient PySpark DataFrame and UDF code.

Structured Streaming Basics

Learn how Spark's Structured Streaming engine treats a live data stream as an unbounded, continuously growing table processed with the same DataFrame API used…

The Catalyst Optimizer

How Spark SQL's Catalyst optimizer moves a query through analysis, logical optimization, physical planning, and code generation, plus how Adaptive Query Execut…

The Spark SQL API

How Spark SQL lets you query DataFrames with standard SQL, compiling to the same Catalyst-optimized plan as the DataFrame API.