Big Data Concepts Cheat Sheet
Explains the core big data concepts, Hadoop/Spark ecosystem components, and columnar storage formats used to process datasets at distributed scale.
2 PagesIntermediateMar 18, 2026
Distributed Processing with PySpark
Aggregate a large dataset across a cluster.
python
from pyspark.sql import SparkSessionfrom pyspark.sql import functions as Fspark = SparkSession.builder.appName("sales-analysis").getOrCreate()df = spark.read.parquet("s3://bucket/sales/")result = ( df.filter(F.col("amount") > 0) .groupBy("region") .agg(F.sum("amount").alias("total_sales"), F.count("*").alias("num_orders")) .orderBy(F.desc("total_sales")))result.write.mode("overwrite").parquet("s3://bucket/output/")spark.stop()
Core Concepts (The Vs)
What distinguishes 'big data' from a regular dataset.
- Volume- Scale of data, often terabytes to petabytes, exceeding single-machine storage/processing
- Velocity- Speed at which data is generated and must be processed (batch vs. streaming)
- Variety- Mix of structured, semi-structured (JSON, XML), and unstructured data (text, images)
- Veracity- Data quality and trustworthiness; noisy or inconsistent data undermines analysis
- Horizontal scaling- Adding more commodity machines to a cluster rather than upgrading a single machine
- Data locality- Moving computation to where the data resides rather than moving data across the network
Ecosystem Components
The building blocks of a typical big data stack.
- HDFS- Hadoop Distributed File System; splits files into blocks replicated across cluster nodes
- MapReduce- Programming model that processes data in parallel Map (transform) then Reduce (aggregate) phases
- Apache Spark- In-memory distributed processing engine, much faster than MapReduce for iterative workloads
- YARN- Yet Another Resource Negotiator; manages cluster resources and job scheduling in Hadoop
- Apache Kafka- Distributed event streaming platform for high-throughput publish/subscribe messaging
- Data lake vs. data warehouse- Data lakes store raw data in any format; warehouses store structured, modeled data
Controlling Partitions
Right-size partitions to balance parallelism against overhead.
python
df = spark.read.parquet("s3://bucket/events/")print(df.rdd.getNumPartitions())# Too many small partitions -> coalesce (no shuffle, merges partitions)df_coalesced = df.coalesce(50)# Too few large / skewed partitions -> repartition (full shuffle)df_repartitioned = df.repartition(200, "customer_id")# Write partitioned by a low-cardinality column to enable partition pruning on readdf.write.partitionBy("event_date").mode("overwrite").parquet("s3://bucket/output/")
Broadcast Joins
Avoid shuffling a huge table by broadcasting the small side of a join.
python
from pyspark.sql.functions import broadcastlarge_df = spark.read.parquet("s3://bucket/events/") # billions of rowssmall_df = spark.read.parquet("s3://bucket/countries/") # a few hundred rows# Ships small_df to every executor instead of shuffling large_df across the clusterresult = large_df.join(broadcast(small_df), "country_code")
Structured Streaming with Watermarks
Process an unbounded Kafka stream with windowed aggregation and late-data handling.
python
from pyspark.sql.functions import windowstream_df = ( spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "broker:9092") .option("subscribe", "clickstream") .load())windowed = ( stream_df .withWatermark("event_time", "10 minutes") # drop data later than 10 min late .groupBy(window("event_time", "5 minutes"), "page") .count())query = ( windowed.writeStream .outputMode("append") .format("console") .trigger(processingTime="1 minute") .start())query.awaitTermination()
Shuffle, Skew & the Query Engine
Why distributed jobs get slow, and what Spark does internally.
- Shuffle- Redistributing data across partitions for wide transforms (groupBy, join); expensive due to disk and network I/O
- Data skew- Uneven key distribution means some tasks process far more data than others and become stragglers
- Salting- Appending a random suffix to skewed keys to spread them across more partitions before a final re-aggregation
- Catalyst optimizer- Spark SQL's planner; applies predicate pushdown, constant folding, and join reordering before execution
- Tungsten- Spark's off-heap memory manager and whole-stage code generation engine for CPU-efficient execution
- Spill- When a partition's working set exceeds executor memory, Spark writes intermediate data to disk, slowing the stage
Lakehouse Table Formats
Open table formats that add database-like guarantees on top of files in object storage.
- Delta Lake- Open table format adding ACID transactions and a versioned transaction log on top of Parquet files
- Apache Iceberg- Table format with hidden partitioning and safe schema/partition evolution without rewriting existing data
- Apache Hudi- Table format optimized for upsert-heavy, incremental ingestion such as streaming CDC into a lake
- Time travel- Query a table as of a previous version or timestamp using the transaction log's history
- ACID on object storage- Atomic commits and snapshot isolation let concurrent readers and writers avoid seeing corrupt or partial data
- Schema evolution- Adding, renaming, or dropping columns without a full rewrite of already-written data files
Pro Tip
Prefer columnar formats like Parquet over CSV/JSON for analytical workloads - reading only the columns you query, combined with predicate pushdown, can cut I/O by an order of magnitude.
Was this cheat sheet helpful?
Explore Topics
#BigDataConcepts#BigDataConceptsCheatSheet#DataScience#Intermediate#DistributedProcessingWithPySpark#CoreConceptsTheVs#EcosystemComponents#ControllingPartitions#MachineLearning#CheatSheet#SkillVeris
