Time Series Basics
pandas was originally built for financial time-series analysis, and its time-handling machinery remains one of its strongest features. At the core are two types: Timestamp, a single point in time (pandas' enhanced replacement for Python's datetime), and DatetimeIndex, an index made up of Timestamps that lets a DataFrame or Series be indexed, sliced, and aggregated by date and time. Converting a column of date strings into real datetime objects with pd.to_datetime() — rather than leaving them as plain strings — unlocks date arithmetic, calendar-aware filtering, resampling, and correct chronological sorting, all of which are unavailable or error-prone on string columns.
Cricket analogy: Converting a column of match-date strings into real Timestamps with pd.to_datetime() unlocks chronological sorting of a career's innings and calendar-aware filtering like 'every match played in 2023', which is unreliable when dates are left as plain text.
Creating and Parsing Datetimes
pd.to_datetime() is the standard entry point for converting strings, lists, or an entire column into Timestamps; it auto-infers common formats but accepts an explicit format string (e.g. format='%d/%m/%Y') for speed and correctness when the format is known and non-standard. pd.date_range(start, end, freq) generates an evenly spaced DatetimeIndex — freq='D' for daily, 'W' for weekly, 'M' for month-end, 'MS' for month-start, 'H' for hourly, and so on — which is invaluable for building a complete calendar skeleton to reindex sparse data against, ensuring no dates are silently missing.
Cricket analogy: pd.to_datetime(dates, format='%d/%m/%Y') parses a scorecard's day-first date strings correctly and quickly, and pd.date_range('2026-01-01', '2026-12-31', freq='D') builds a full calendar of match days to reindex a sparse fixture list against, exposing missing dates.
import pandas as pd
raw = pd.DataFrame({
'date_str': ['2026-01-05', '2026-01-06', '2026-01-08'],
'sales': [120, 95, 130]
})
raw['date'] = pd.to_datetime(raw['date_str'])
raw = raw.set_index('date')
# a complete daily calendar, including days with no recorded sales
full_range = pd.date_range('2026-01-05', '2026-01-08', freq='D')
raw_reindexed = raw.reindex(full_range)
print(raw_reindexed)
# date_str sales
# 2026-01-05 2026-01-05 120.0
# 2026-01-06 2026-01-06 95.0
# 2026-01-07 NaN NaN
# 2026-01-08 2026-01-08 130.0
# datetime attribute access via .dt accessor
raw['weekday'] = raw.index.day_name()
print(raw[['sales', 'weekday']])Date-Based Indexing and the .dt Accessor
Once a DataFrame has a DatetimeIndex, powerful partial-string indexing becomes available: df.loc['2026-01'] selects every row in January 2026, and df.loc['2026-01-05':'2026-01-08'] performs an inclusive date-range slice, both far more concise than manually comparing timestamp bounds. For datetime values stored in a regular column rather than the index, the .dt accessor exposes the same component extraction — .dt.year, .dt.month, .dt.day_name(), .dt.is_month_end — mirroring how .str works for string columns.
Cricket analogy: df.loc['2026-01'] on a scorecard with a DatetimeIndex instantly selects every match played in January 2026, and .dt.day_name() on a match-date column reveals whether a Test started on a Thursday, mirroring how .str works for text columns.
Timestamps carry an optional timezone. tz_localize() attaches a timezone to naive (timezone-unaware) timestamps, while tz_convert() converts an already timezone-aware Timestamp to a different timezone — mixing naive and aware timestamps in comparisons raises an error, a common source of confusion when merging data from different sources.
Sorting matters: many time-series operations (especially resampling and rolling windows, covered elsewhere) assume the DatetimeIndex is sorted ascending. Always call df.sort_index() after setting a datetime index built from unsorted source data.
- pd.to_datetime() converts strings or columns into proper Timestamp objects, enabling date arithmetic and correct sorting.
- pd.date_range() generates a regular DatetimeIndex with a chosen frequency (D, W, M, H, etc.), useful for building a complete calendar skeleton.
- A DatetimeIndex enables partial-string indexing like df.loc['2026-01'] to select an entire month.
- The .dt accessor extracts datetime components (year, month, day_name, is_month_end) from a datetime column.
- Timestamps can be timezone-naive or timezone-aware; tz_localize() attaches a timezone, tz_convert() changes it.
- A DatetimeIndex should generally be sorted ascending before performing time-based slicing, resampling, or rolling operations.
Practice what you learned
1. What is the primary purpose of pd.to_datetime()?
2. What does pd.date_range('2026-01-01', '2026-01-10', freq='D') produce?
3. Given a DataFrame with a DatetimeIndex, what does df.loc['2026-01'] select?
4. What is the .dt accessor used for?
5. What happens if you try to compare a timezone-naive Timestamp with a timezone-aware Timestamp?
Was this page helpful?
You May Also Like
Resampling and Rolling Windows
Master resample() for changing time-series frequency and rolling()/expanding() for moving-window calculations like moving averages.
Working with Dates and Times
Core techniques for parsing, storing, and manipulating temporal data in pandas using Timestamp, datetime64 columns, the .dt accessor, and Timedelta arithmetic.
Sorting and Ranking
How to order DataFrames and Series with sort_values and sort_index, and how to assign competitive ranks to values with the rank method and its tie-breaking strategies.
Handling Missing Data in Pandas
Learn how pandas represents missing values with NaN and NA, and the core toolkit — isna, dropna, fillna, interpolate — for detecting and handling them.