Concatenating DataFrames
Concatenation stacks multiple pandas objects together along an axis, without matching on key columns the way merge does. It's the tool of choice when you have several DataFrames that share the same structure — monthly sales files, per-region exports, batched query results — and you simply want to stack them into one combined DataFrame. pd.concat is more general than merge: it can join any number of objects at once, works on both Series and DataFrames, and can combine along either rows (axis=0, the default) or columns (axis=1).
Cricket analogy: Stacking monthly scorecards from IPL, ODI, and Test matches into one master sheet with pd.concat just piles rows together without matching player names like a merge would, and works whether you're combining two formats or twenty.
Stacking rows with axis=0
The most common use of concat is appending rows: pd.concat([df1, df2, df3]) stacks the DataFrames vertically, preserving each frame's original index by default (which can create duplicate index labels across the combined result). Passing ignore_index=True generates a fresh RangeIndex for the combined DataFrame instead, which is usually what you want when the original indices carry no meaning. If the DataFrames have different columns, concat aligns by column name, filling missing columns with NaN unless join='inner' is specified to keep only shared columns.
Cricket analogy: pd.concat([jan, feb, mar]) stacks three monthly scorecards vertically but keeps each sheet's original row numbers, risking duplicates; ignore_index=True renumbers cleanly, and if march added a 'strike_rate' column, missing values in jan/feb fill with NaN unless join='inner' keeps only shared columns.
import pandas as pd
jan = pd.DataFrame({'region': ['North', 'South'], 'sales': [100, 150]})
feb = pd.DataFrame({'region': ['North', 'South'], 'sales': [120, 90]})
combined = pd.concat([jan, feb], keys=['Jan', 'Feb'], ignore_index=False)
print(combined)
# region sales
# Jan 0 North 100
# 1 South 150
# Feb 0 North 120
# 1 South 90
combined_flat = pd.concat([jan, feb], ignore_index=True)
print(combined_flat.index.tolist()) # [0, 1, 2, 3]Combining columns with axis=1
pd.concat([df1, df2], axis=1) instead lines objects up side by side, aligning rows by their index. This is useful for adding a set of pre-computed columns back onto an existing DataFrame, provided the indices correspond correctly. Because alignment happens on the index, mismatched or unsorted indices between the two frames can produce unexpected NaN-filled rows — it's good practice to ensure the indices match (or are both reset) before an axis=1 concat.
Cricket analogy: pd.concat([batting_stats, bowling_stats], axis=1) lines up a player's runs and wickets side by side by matching their index; if the bowling sheet was re-sorted first, mismatched row order produces NaN-filled columns instead of correctly paired stats.
concat versus merge
concat and merge solve different problems. concat is a structural stacking operation — it does not match on key values, only on the shared axis (index for axis=1, or column names for axis=0's alignment of differing columns). merge, by contrast, performs relational key-based matching, similar to a SQL join, and can change row counts based on how keys align between the two objects. If you need to combine tables based on a shared identifier like customer_id, use merge; if you're stacking data with an already-compatible shape or index, use concat.
Cricket analogy: Stacking two identical-format innings sheets is a concat job, just piling rows together; but combining a batting sheet with a separate fielding sheet by matching player_id is a merge job, matching keys like a relational join.
The keys parameter in concat creates a hierarchical (MultiIndex) label for each source object, letting you trace every row in the combined result back to which original DataFrame it came from — extremely useful for debugging batch-loaded data.
Concatenating many DataFrames one at a time inside a Python loop with repeated pd.concat calls is quadratic in cost. Collect all the pieces into a list first and call pd.concat(list_of_dfs) once for much better performance.
- pd.concat stacks multiple Series/DataFrames along an axis without key-based matching, unlike merge.
- axis=0 (default) stacks rows; axis=1 lines objects up side by side, aligning on the index.
- ignore_index=True produces a fresh RangeIndex instead of preserving (and possibly duplicating) original indices.
- join='outer' (default) keeps all columns/rows, filling NaN for missing ones; join='inner' keeps only the shared ones.
- The keys parameter labels each source object in a resulting MultiIndex, aiding traceability.
- Building a list of DataFrames and calling concat once is far more efficient than repeated concat calls in a loop.
Practice what you learned
1. What is the default axis along which pd.concat combines a list of DataFrames?
2. What problem does ignore_index=True solve when concatenating several DataFrames that each have a default RangeIndex?
3. How does pd.concat with axis=1 align the DataFrames being combined?
4. What is the key structural difference between pd.concat and pd.merge?
5. Why is calling pd.concat once on a list of DataFrames preferred over calling it repeatedly inside a loop?
Was this page helpful?
You May Also Like
Merging and Joining DataFrames
How pandas' merge function combines two DataFrames on shared keys using inner, left, right, and outer join logic, and how it compares to the index-based join method.
DataFrame Basics
Understand the structure of a pandas DataFrame — rows, columns, index, and dtypes — and the core operations for creating, inspecting, and selecting from tabular data.
Reading and Writing Data
Learn pandas' file I/O functions — read_csv, read_excel, read_json, to_csv, and to_parquet — plus the key parameters that control parsing, dtypes, and output format.
MultiIndex Basics
Learn how pandas' hierarchical MultiIndex lets you represent and query higher-dimensional data using multiple index levels on a single axis.