Dask Cheat Sheet
Dask reference covering parallel DataFrame and Array APIs, delayed task graphs, lazy evaluation, and the distributed scheduler client.
2 PagesIntermediateMar 28, 2026
Dask DataFrame
Pandas-like API that scales out of core.
python
import dask.dataframe as dddf = dd.read_csv("data-*.csv") # reads many files as one logical DataFrameresult = df.groupby("category")["value"].mean()computed = result.compute() # triggers actual executiondf["ratio"] = df["a"] / df["b"] # lazy, builds a task graphdf.to_parquet("output/", engine="pyarrow")
Dask Array & Delayed
Parallelize NumPy-style code and arbitrary functions.
python
import daskfrom dask import delayedimport dask.array as dax = da.random.random((10000, 10000), chunks=(1000, 1000))y = (x + x.T).mean(axis=0)result = y.compute()@delayeddef process(x): return x * 2tasks = [process(i) for i in range(10)]results = dask.compute(*tasks)
Distributed Client
Run tasks on a local or remote cluster.
python
from dask.distributed import Clientclient = Client(n_workers=4, threads_per_worker=2) # local clusterprint(client.dashboard_link) # web UI for the task graphfuture = client.submit(process, 10)result = future.result()client.close()
Core Concepts
Fundamental Dask building blocks.
- dask.dataframe- parallel pandas-like API for larger-than-memory tables
- dask.array- parallel NumPy-like API for chunked arrays
- dask.delayed- wraps arbitrary Python functions into lazy task graphs
- .compute()- triggers execution and returns a concrete result
- Client- connects to a local or distributed scheduler with a diagnostics dashboard
- chunks / partitions- controls parallelism granularity for arrays/dataframes
persist(), optimize(), and Graph Introspection
Pin intermediate results in distributed memory and inspect the optimized task graph before computing.
python
from dask.distributed import Clientfrom dask import optimizeimport dask.dataframe as ddclient = Client()df = dd.read_parquet("s3://bucket/data/*.parquet")filtered = df[df["value"] > 0]# persist() runs the graph now and keeps results in distributed memory,# unlike compute() which also pulls everything back to the clientfiltered = filtered.persist()client.rebalance(filtered) # spread partitions evenly across workers# Fuse and simplify the graph without executing it(optimized_df,) = optimize(filtered)print(len(optimized_df.dask)) # number of tasks in the graph# Visualize the task graph (writes graph.png, requires graphviz)filtered.visualize(filename="graph.png", optimize_graph=True)
Custom Graphs & Task Annotations
Build raw task graphs and control scheduling priority, resources, and retries per task.
python
import daskfrom dask import get# A task graph is just a dict of {key: (callable, *args)}graph = { "load": (open, "data.csv"), "parse": (lambda f: f.read().splitlines(), "load"), "count": (len, "parse"),}result = get(graph, "count")# Annotate delayed tasks with priority, GPU resource constraints, and retrieswith dask.annotate(priority=10, retries=3, resources={"GPU": 1}): fut = dask.delayed(train_model)(params)# Workers must be launched with matching resource tags, e.g.# dask-worker scheduler:8786 --resources "GPU=2"
map_partitions / map_blocks with Metadata
Apply arbitrary functions per-partition or per-chunk while declaring output dtypes so Dask can build the graph lazily.
python
import pandas as pdimport dask.dataframe as ddimport dask.array as daimport numpy as npdef enrich(partition: pd.DataFrame) -> pd.DataFrame: partition["z"] = (partition["x"] - partition["x"].mean()) / partition["x"].std() return partition# meta tells Dask the output schema without running the function eagerlyresult = df.map_partitions(enrich, meta=df._meta.assign(z=np.float64()))# Array equivalent: apply a NumPy function chunk-by-chunkx = da.random.random((10000,), chunks=1000)squared = x.map_blocks(lambda block: block ** 2, dtype=x.dtype)
Shuffling & Repartitioning for Joins
Control partition count and shuffle method to avoid memory blowups on merges and set_index.
python
import dask.dataframe as dd# set_index triggers a full shuffle — sort=True enables fast loc/merge laterdf = df.set_index("user_id", sorted=False, shuffle="tasks")# repartition before an expensive groupby/merge to right-size partitionsdf = df.repartition(npartitions=200)# merges on the index avoid a shuffle when both sides are index-sortedmerged = df.merge(other_df, left_index=True, right_index=True)# p2p shuffle (distributed scheduler) is far more memory-efficient than# the default task-based shuffle for large joinsmerged = df.merge(other_df, on="user_id", shuffle_method="p2p")
Scheduler & Cluster Tuning
Knobs that matter once a workload moves past a laptop-scale dataset.
- scheduler="threads" / "processes" / "synchronous"- picks the local scheduler backend; synchronous is invaluable for debugging with pdb
- client.scatter(data, broadcast=True)- pushes large read-only data to all workers once instead of re-serializing it per task
- worker_class / nthreads / memory_limit- per-worker resource caps passed to LocalCluster or dask-worker to prevent OOM kills
- distributed.utils.spill-to-disk- workers automatically spill to disk under memory pressure; tune via distributed.yaml
- Adaptive scaling (cluster.adapt())- scales worker count between minimum/maximum based on queued task load, e.g. on Kubernetes/YARN
- fuse / optimize_graph- graph-optimization passes that merge chained tasks to cut scheduler overhead
- dask.config.set- programmatically overrides settings like array chunk-size or dataframe shuffle-compression
Pro Tip
Dask operations are lazy by default — build up your full pipeline of transformations first, then call .compute() once at the end so Dask can optimize and parallelize the whole task graph instead of materializing intermediate results.
Was this cheat sheet helpful?
Explore Topics
#Dask#DaskCheatSheet#DataScience#Intermediate#DaskDataFrame#DaskArrayDelayed#DistributedClient#CoreConcepts#DataStructures#MachineLearning#APIs#Concurrency#CheatSheet#SkillVeris
