Polars (DataFrame Library) Cheat Sheet
Manipulate large tabular datasets fast with Polars' expression API, lazy execution, and multi-threaded query engine in Python and Rust.
Read, Select, and Filter
Load a Parquet file and perform basic column selection and row filtering.
import polars as pldf = pl.read_parquet("orders.parquet")result = df.select(["order_id", "amount", "status"]).filter( (pl.col("amount") > 100) & (pl.col("status") == "completed"))print(result.head())
Lazy Query with the Expression API
Build a query plan with LazyFrame and let the optimizer fuse operations before execution.
lf = ( pl.scan_parquet("orders.parquet") .filter(pl.col("status") == "completed") .group_by("customer_id") .agg([ pl.col("amount").sum().alias("total_spent"), pl.col("order_id").count().alias("n_orders"), ]) .sort("total_spent", descending=True))result = lf.collect() # optimizer runs hereprint(lf.explain()) # inspect the query plan
Joins and Window Functions
Join two frames and compute a per-group rolling calculation.
joined = orders.join(customers, on="customer_id", how="left")with_rank = joined.with_columns( pl.col("amount").rank(descending=True).over("customer_id").alias("spend_rank"), pl.col("amount").rolling_mean(window_size=3).over("customer_id").alias("rolling_avg"),)
Interop with pandas and Arrow
Convert between Polars, pandas, and Arrow without copying more than necessary.
pandas_df = df.to_pandas()polars_df = pl.from_pandas(pandas_df)# zero-copy-ish conversion via Arrowarrow_table = df.to_arrow()df2 = pl.from_arrow(arrow_table)
Common Expressions
Frequently used building blocks inside select/filter/with_columns.
- pl.col("x")- references a column inside an expression
- pl.lit(value)- inserts a literal scalar into an expression
- .over("group_col")- turns an expression into a windowed/group-wise calculation
- .alias("name")- renames the result of an expression
- pl.when(cond).then(a).otherwise(b)- vectorized conditional logic
- df.lazy() / lf.collect()- switch between eager and lazy execution modes
Streaming Execution for Larger-Than-RAM Data
Process datasets bigger than memory by streaming batches through the query engine instead of materializing the full result.
lf = ( pl.scan_csv("huge_events_*.csv") .filter(pl.col("event_type") == "purchase") .group_by("country") .agg(pl.col("revenue").sum()))# streaming=True processes the plan in chunks, keeping peak memory lowresult = lf.collect(streaming=True)# or write straight to disk without ever collecting to memorylf.sink_parquet("country_revenue.parquet")
Nested Struct and List Columns
Build and unpack nested data without exploding to a wide, sparse schema.
df = pl.DataFrame({ "id": [1, 2], "tags": [["a", "b"], ["c"]],})# struct: pack multiple columns into one nested columnpacked = df.with_columns( pl.struct(["id", "tags"]).alias("record"))# list ops without a Python-level loopexploded = df.explode("tags")counted = df.with_columns(pl.col("tags").list.len().alias("n_tags"))has_a = df.with_columns(pl.col("tags").list.contains("a").alias("has_a"))
Time-Based Rolling Aggregation
Bucket an irregularly-sampled time series into fixed windows with group_by_dynamic.
events = events.sort("ts")windowed = events.group_by_dynamic( "ts", every="1h", period="1h", closed="left", by="sensor_id").agg([ pl.col("value").mean().alias("avg_value"), pl.col("value").max().alias("peak_value"),])
Run SQL Against LazyFrames
Register frames under names and query them with plain SQL when it's more concise than the expression API.
ctx = pl.SQLContext(orders=orders_lf, customers=customers_lf, eager=False)result = ctx.execute(""" SELECT c.region, SUM(o.amount) AS total FROM orders o JOIN customers c ON o.customer_id = c.customer_id GROUP BY c.region ORDER BY total DESC""").collect()
Advanced Expression Toolkit
Less common but powerful building blocks for expression pipelines.
- pl.col("^price_.*$")- selects columns by regex pattern instead of listing names explicitly
- .pipe(fn)- threads a DataFrame/LazyFrame through a custom function while staying in a fluent chain
- .map_batches(fn, return_dtype=...)- applies a vectorized (not row-wise) custom function inside an expression, far faster than map_elements
- pl.concat_str([...], separator="-")- concatenates multiple columns into one string column
- df.partition_by("key")- splits a DataFrame into a list of DataFrames, one per distinct key value
- pl.Config.set_fmt_str_lengths(n)- controls how much of long string cells is shown when printing, useful for debugging wide text columns
Default to scan_parquet + LazyFrame chains instead of read_parquet + eager DataFrame — the query optimizer can push filters and column projections down into the file scan, often cutting memory use by an order of magnitude on wide files.
