Dates and Time Series with pandas

A date is more useful when pandas knows it is a date. A time series ties observations to moments or spans of time. Hourly energy readings form a regular series; orders arriving at uneven intervals form an irregular one. Text such as "12/31/2024" can sort after "03/01/2025", even though December came first. Consistent full ISO strings sort correctly, but datetime values also support calendar selection and arithmetic.

Choose the right time object

Python supplies datetime, date and timedelta. In pandas, a Timestamp holds one moment, a datetime column has a datetime64[unit] data type, and a DatetimeIndex holds datetime row labels. NaT means a missing time. Detect it with pd.isna(); subtracting a missing time produces another missing time.

The unit in datetime64[us] is microseconds, the storage resolution. Other units include seconds (s), milliseconds (ms) and nanoseconds (ns). pandas 3 normally parses text at microsecond resolution, using nanoseconds when the text needs them. Finer resolution uses more of the integer range: nanosecond dates extend only from 1677 to 2262.

Calculate elapsed time

import pandas as pd
start = pd.Timestamp("2025-03-10 18:00")
end = pd.Timestamp("2025-03-14 09:30")
elapsed = end - start
hours = elapsed / pd.Timedelta(hours=1)

elapsed is 3 days, 15 hours and 30 minutes. The three days contribute 72 hours, so hours is 87.5. elapsed.total_seconds() gives 315,000. Its .days attribute gives only 3; it is the whole-day component, not the complete duration in days. Build durations with pd.Timedelta(hours=36) or parse several with pd.to_timedelta(["1h", "90min"]).

Subtracting a timestamp from a whole datetime column produces durations row by row. Use .dt.days for whole-day components or divide the duration column by a unit Timedelta for fractional totals.

Read calendar parts and round times

On a column, .dt.year, .dt.month, .dt.hour and .dt.quarter extract calendar parts. .dt.day_of_week numbers Monday as 0 through Sunday as 6; .dt.day_name() gives a name. March 14, 2025 is Friday, numbered 4. An index uses these attributes directly: df.index.month. The spellings day_of_week and day_of_year replace older spellings deprecated in pandas 3.1.

A weekday can become a grouping key: df.groupby(df["timestamp"].dt.day_name())["kwh"].mean(). Grouping by month number combines that month across years; a monthly period keeps each year separate.

For 09:37, .dt.floor("h") gives 09:00, .dt.ceil("h") gives 10:00, and .dt.round("15min") gives 09:30. .dt.normalize() sets clocks to midnight while keeping datetime values. .dt.date instead produces Python date objects and loses datetime-column features.

Keep numeric units explicit

astype("int64") returns counts in the column's own unit. A microsecond count is 1,000 times smaller than the nanosecond count for the same moment. If an external calculation needs nanoseconds, use s.dt.as_unit("ns").astype("int64"); a Timestamp or index uses .as_unit("ns") directly. For elapsed time, Timedelta arithmetic avoids that unit trap.

Parsing gives text a calendar meaning. Before converting a date column, establish its format and check whether it contains offsets, invalid entries or missing values. ISO 8601 writes the year first, as in "2025-03-14T09:30:00", and avoids day/month ambiguity.

State the source format

"03/04/2025" can mean March 4 or April 3. pd.to_datetime("03/04/2025") reads month first; dayfirst=True reads day first. A known format is clearer:

raw = pd.Series(["14/03/2025 18:45", "15/03/2025 09:00"])
parsed = pd.to_datetime(raw, format="%d/%m/%Y %H:%M")

%Y is a four-digit year, %m a month and %d a day. %H is a 24-hour clock hour, %M a minute and %S a second. Use %I with %p for a 12-hour clock and AM/PM. %b and %B read abbreviated and full month names. Uppercase %M means minutes, so it cannot replace lowercase %m for months.

Without format, pandas infers a format from the first non-missing entry and applies it to the column. With "03/04/2025" first, a later "13/04/2025" fails under month-first inference. Reordering the rows can change the inferred format. format="ISO8601" handles ISO variants. format="mixed" parses entries individually, but cannot settle an ambiguous source convention; clean or verify that convention first.

Inspect failed conversions

raw = pd.Series(["2025-01-15", "not a date", None])
parsed = pd.to_datetime(raw, format="%Y-%m-%d", errors="coerce")
failed = parsed.isna() & raw.notna()

The result contains January 15 and two NaT values. parsed.isna().sum() is 2 because it includes the original missing entry. failed.sum() is 1 because only "not a date" was present and failed. Inspect raw[failed] before deciding to repair or remove it. Coercion keeps a load running, but also hides typos such as an impossible month. The default errors="raise" stops on invalid input; pandas 3 removed errors="ignore".

Parse files and numeric timestamps

pd.read_csv("meter.csv", parse_dates=["timestamp"], index_col="timestamp") requests datetime parsing and a time index. Check the resulting dtypes after loading. Text columns in pandas 3 use the str dtype; older versions commonly show object.

Join separate date and time strings with a space before parsing. Columns named year, month and day can be assembled directly with pd.to_datetime(df[["year", "month", "day"]]).

For Unix counts, specify the unit. pd.to_datetime(1735689600, unit="s", utc=True) gives midnight UTC on January 1, 2025. The millisecond count is 1735689600000 with unit="ms". Using the wrong unit changes the interpreted moment.

Format only for presentation

parsed.dt.strftime("%d %b %Y") writes text such as "15 Jan 2025". A Timestamp's .isoformat() writes an ISO string. Named months and weekdays follow the system locale; the displayed examples use English. Keep datetimes during analysis and format at export, because strings no longer support date arithmetic.

A useful time index is parsed, ordered and checked. A table can contain several events at one moment, missing days or out-of-order batches. Each condition changes what later calculations mean.

Build datetime row labels

df["timestamp"] = pd.to_datetime(df["timestamp"], format="%Y-%m-%d %H:%M")
df = df.set_index("timestamp").sort_index()

set_index() moves the column into the row labels; it does not parse text. Inspect df.index.dtype for a datetime dtype and df.index.is_monotonic_increasing for chronological order. Equal neighboring times are allowed in an increasing index, so this check does not establish uniqueness.

A DatetimeIndex supports date selection and resampling. Text labels do not become a time index merely because they look like dates. You can also leave timestamps in a column: df.resample("D", on="timestamp") uses that column without changing the index. A PeriodIndex is another time-series representation when rows describe spans such as months.

Decide what repeated times mean

df.index.is_unique reports whether every label is different. df.index.duplicated() marks repeated labels after the first. Suppose two readings, 3 and 5, share January 1 and a reading of 4 belongs to January 2. Averaging duplicate readings with df.groupby(level=0).mean() produces 4.0 on both dates.

That operation is a decision about the data. If rows are separate purchases, summing may be appropriate. If an import duplicated a record, keeping the last copy with df[~df.index.duplicated(keep="last")] may be appropriate. Do not delete valid transactions solely to make an index unique. Operations that place one value at every target label, including reindex() and asfreq(), require unique source labels.

Distinguish stored and inferred frequency

An index created by pd.date_range() normally carries .freq. Parsed timestamps often have .freq equal to None, even when they fall on consecutive days. pd.infer_freq(df.index) examines at least three timestamps and returns an alias such as "D" or "ME" if it finds a regular pattern. Irregular spacing returns None.

Frequency metadata is not a completeness audit. To look for absent daily rows, build the calendar you expect:

calendar = pd.date_range(df.index.min(), df.index.max(), freq="D")
missing = calendar.difference(df.index)

A missing day has no row. It differs from a present row containing NaN. A calendar bounded by the observed minimum and maximum cannot reveal missing dates outside that range; use the planned reporting start and end when those are known.

Prepare for later operations

Sorting makes slices predictable and is required for time-based rolling windows. Missing rows matter even when a calculation succeeds: shift(1) means one row back, and rolling(7) means seven rows. Neither checks that each row represents one day. For a long table, also keep the series key, such as sensor or store, so later operations can run separately per series.

Once the index contains sorted datetimes, date strings become useful selectors. Their precision controls how much time they represent: a year, a month, a day or a particular minute.

Select a whole calendar period

For daily data from January through March 2025, ts.loc["2025"] selects all 90 days. ts.loc["2025-02"] selects February's 28 days. On hourly data, a day string selects that whole day, not just midnight.

Use .loc on a DataFrame too:

february = df.loc["2025-02", "kwh"]
spring_onward = df.loc["2025-03-01":]

df["2025-02"] instead asks for a column with that name. The presence of a time index does not change square-bracket column selection into row selection.

Read the endpoint precisely

A label slice includes both endpoints. ts.loc["2025-01-30":"2025-02-02"] includes January 30 and 31 and February 1 and 2. A position slice such as ts.iloc[0:4] stops before position 4.

The end of a partial-string slice includes the entire named period. On hourly data, df.loc[:"2025-06-01"] includes all of June 1. df.loc[:"2025-06-01 00:00"] stops at midnight. Likewise a slice ending in "2025-02" includes the whole of February. Choosing a more precise string changes the boundary, so write the cutoff to match the question.

ts.truncate(before="2025-03-01") keeps March 1 onward. Its after= argument trims the other end. Sorting first avoids failures that can occur when date-string slices meet unordered labels.

Select recurring calendar conditions

Calendar attributes produce boolean masks:

weekends = df[df.index.day_of_week >= 5]
evening_march = df[(df.index.month == 3) & (df.index.hour >= 18)]

Monday is 0, so Saturday and Sunday are 5 and 6. Each condition needs parentheses when combined with & or |. With a datetime column, use df["timestamp"].dt.day_of_week instead of an index attribute. The weekday mask selects a repeated calendar pattern; it is different from selecting one continuous date interval.

Select a clock window on every date

df.between_time("09:00", "17:00") selects those hours each day. Both ends are included by default, giving nine rows in a complete hourly day. inclusive="left" includes 09:00 and excludes 17:00, giving eight. df.at_time("12:00") selects exactly noon each day.

A clock window can cross midnight. between_time("22:00", "06:00") selects late-night and early-morning rows across the index. It does not select one uninterrupted interval between two particular dates. On an aware index, date-string selection uses the index's local zone; convert the index before selecting a reporting day in a different zone.

A change needs a clear comparison point. "Previous" can mean the previous observed row, the previous calendar day or the same month a year ago. Choose that meaning before computing a lag or growth rate.

Put earlier values beside current ones

For visitor counts 200, 210 and 189 on March 3–5, s.shift(1) gives NaN, 200 and 210. It moves values down one row while keeping the labels. The new first slot is missing because no earlier observation exists. shift(-1) gives 210, 189 and NaN: a lead that brings the next value to the current row.

df["previous"] = df["visitors"].shift(1)
df["change"] = df["visitors"].diff()
df["growth"] = df["visitors"].pct_change()

diff() subtracts the preceding value, giving NaN, 10 and −21. pct_change() divides the current value by the previous one and subtracts 1, giving NaN, 0.05 and −0.10. These are fractions: 0.05 means 5%, not 0.05%. Floating-point output can have tiny extra digits; round for display after calculating.

Choose the lag in rows or in time

pct_change(7) compares with seven rows earlier. It is the same weekday a week earlier only when there is exactly one row per day with no gaps. On a complete monthly series, pct_change(12) gives year-over-year change.

shift(1, freq="D") moves the timestamps forward one day and leaves the values attached to those moved labels. It keeps every value. Arithmetic with the original then aligns by date:

previous_day = s.shift(1, freq="D")
daily_change = s - previous_day

If the source has observations on Monday and Thursday only, plain shift(1) compares Thursday with Monday. A date shift offers a comparison with Wednesday; if Wednesday is absent, Thursday's daily change is missing. Filling that gap is a separate decision about the data.

Expect arithmetic to align labels

Two Series combine by timestamp, not by row position. Add values 200, 210 and 189 on March 3, 4 and 5 to values 1 and 1 on March 4 and 7. Only March 4 has two observations, so its sum is 211. The union of dates is March 3, 4, 5 and 7; the other results are missing.

Use s.add(other, fill_value=0) when an absent observation genuinely means zero, such as a verified event count. It is not a general repair for missing sensor readings.

Restart comparisons per series

Sort a long table by its series key and date, then calculate within groups. df.groupby("store")["sales"].pct_change() compares each store only with itself. For store A selling 100 then 110 and store B selling 50 then 40, the growth rates are missing, 0.10, missing and −0.20. A plain column-wide calculation would compare B's first sale with A's last sale. That numerical result would describe a switch of stores, not growth over time.

A regular date range is a calendar you can generate instead of typing every label. It can label test data, define an expected reporting grid or reveal missing observations after reindexing.

Specify a start and a rule

ends = pd.date_range("2025-01-01", periods=4, freq="ME")

The four timestamps are January 31, February 28, March 31 and April 30. "ME" means month end, so the first date is the first month end on or after the start. A regular frequency can follow a calendar rule even when its elapsed gaps vary in length.

Normally give two of start, end and periods, plus freq. Without an explicit frequency, the default is one calendar day. periods is a count of timestamps, not a count of gaps: four hourly timestamps contain three gaps.

Read common aliases

Multiples express wider intervals: "15min", "2h" and "90min". Starting at 08:00 with four "90min" periods gives 08:00, 09:30, 11:00 and 12:30. "1h30min" expresses the same spacing.

pandas 3 timestamp frequencies use "ME", "QE", "YE", "BME", "h", "min", "s", "ms" and "us". Older aliases "M", "Q", "Y" or "A", "BM", "H", "T", "S", "L" and "U" no longer work in these timestamp operations. Monthly periods still use "M" because a period describes a span.

Follow anchors and business days

An anchored range does not force the supplied start into its output. Starting January 15, 2025 with freq="MS" gives February 1 first. In January 2025, "W-MON" gives January 6, 13, 20 and 27; plain "W" gives Sundays. "WOM-3FRI" means the third Friday of each month.

"B" skips weekends, but it does not know local holidays. March 2025 contains 21 Monday–Friday dates. A custom calendar can exclude a supplied holiday:

pd.bdate_range("2025-03-01", "2025-03-31",
               freq="C", holidays=["2025-03-17"])

That range contains 20 dates. The holiday list is explicit; a country is not inferred from a time zone or language.

Count endpoints carefully

Matching start and end points are included by default. Hourly from January 1 at midnight through January 2 at midnight gives 25 timestamps. Use periods=24 for exactly 24 hourly readings, or inclusive="left" to exclude the end. normalize=True resets the start and end clocks to midnight. Add tz="Asia/Tokyo" when the generated labels must carry a zone.

"A month later" is a calendar instruction. "Thirty days later" is elapsed time. The results can differ because months and years do not all contain the same number of days.

Separate duration from calendar movement

A Timedelta adds a fixed amount of elapsed time. A DateOffset follows calendar fields. From January 31, 2025:

t = pd.Timestamp("2025-01-31")
t + pd.DateOffset(months=1)   # February 28
t + pd.Timedelta(days=30)    # March 2

For the DateOffset, first move to February, then keep day 31 if it exists. It does not, so pandas clips to February's last valid day. From January 31, 2024, the result is February 29 because it is a leap year. pd.Timedelta(months=1) is invalid: a month has no single fixed duration.

The same distinction applies to years. From January 1, 2024, 365 elapsed days end on December 31, 2024. pd.DateOffset(years=1) ends on January 1, 2025. Use the unit named by the real question instead of substituting a convenient number of days.

Move to a calendar anchor

pd.offsets.MonthEnd(1) moves to the next month end. Starting January 15, 2025 gives January 31. Starting on January 31 gives February 28. The count of one means advance to an anchor, so an already matching date moves too.

MonthEnd(0) moves a date forward to its current month end but leaves an existing month end unchanged. It is useful for labeling every observation by the end of its month:

month_labels = df.index + pd.offsets.MonthEnd(0)

An offset also has rollforward() and rollback(). For January 15, a MonthEnd's nearest forward anchor is January 31 and its nearest backward anchor is December 31, 2024. These operations select the nearest valid anchor in a direction.

Move through business days

March 14, 2025 is Friday. Adding pd.offsets.BDay(1) gives Monday, March 17. Adding one elapsed day gives Saturday, March 15. A standard business-day rule skips weekends, with holidays requiring an explicitly supplied custom calendar.

Frequency strings are short names for offset objects: "ME" corresponds to MonthEnd, "B" to BusinessDay and "W-MON" to a Monday-anchored week. This lets the same calendar meaning drive a range, resampling rule or date shift.

Keep the intended clock across daylight saving time

In pandas 3, pd.offsets.Day(1) means one calendar day and keeps the local clock time. pd.Timedelta(days=1) adds exactly 24 hours. From noon on March 29, 2025 in Amsterdam, the calendar offset ends at noon March 30, after only 23 elapsed hours. The Timedelta ends at 13:00. A recurring local appointment and a measured equipment runtime therefore need different operations.

A monthly sales total belongs to a whole month. Labeling it with one timestamp is possible, but a Period states its span directly. A period combines a value with a frequency, so "2025-03" with frequency "M" means all of March 2025.

Represent and move spans

march = pd.Period("2025-03", freq="M")
months = pd.period_range("2025-01", "2025-04", freq="M")

months is a four-item PeriodIndex with dtype period[M]. March's start_time is March 1 at midnight; its end_time is March 31 at 23:59:59.999999 at microsecond resolution. An end is the final moment, not midnight at the start of the final day.

Period arithmetic moves in the period's own unit. march + 1 is April 2025. June minus March at monthly frequency gives an offset of three month ends, whose .n is 3. A monthly period also knows .days_in_month: February 2025 has 28 days.

Convert moments to spans and back

monthly_labels = df.index.to_period("M")
column_labels = df["date"].dt.to_period("M")
start_labels = monthly_labels.to_timestamp()
end_labels = monthly_labels.to_timestamp(how="end")

Every timestamp in March maps to the same monthly period. Converting that period back gives its start by default or its end when requested. This does not recover an observation's original day and clock time: the conversion to a period discarded that detail.

Keep the alias families separate. A monthly span uses "M" in Period and period_range. A month-end timestamp uses "ME" in date_range and datetime resampling. The two names describe different kinds of labels.

Change a period's frequency

A period's .asfreq() selects a containing or contained period at a different frequency. march.asfreq("D", how="start") gives March 1 as a daily period; how="end" gives March 31. A yearly period converted to "M" with how="start" gives January.

This selects a boundary period; it does not distribute a monthly total among days. For a PeriodIndex series, resampling can aggregate monthly spans to quarterly spans. Counts of the days in each month of 2025 sum to quarter totals 90, 91, 92 and 92.

Read fiscal quarter labels

A fiscal quarter is named within a fiscal year whose end month is explicit. With "Q-JUN", fiscal 2025 ends in June 2025. Its first quarter therefore runs July–September 2024:

q = pd.Period("2025Q1", freq="Q-JUN")

With "Q-MAR", May 10, 2025 falls in 2026Q1 because that fiscal year ends in March 2026. The year in a fiscal label need not match the calendar year of its first day. For timestamp filters, selecting times before the next period starts is often clearer than constructing the last fractional second of the current one.

Resampling answers questions such as "How much energy was used each day?" or "What was the average temperature each month?" It groups observations into time bins and then applies an aggregation.

Choose a statistic that fits the quantity

daily = df.resample("D").agg({"kwh": "sum", "temp_c": "mean"})

Energy in kWh and rainfall in mm add over time. Temperature is a level, so a mean describes it more usefully than a sum. last() suits a closing balance, while max() finds a peak. Daily sales can be summed for monthly total sales or averaged for average sales per observed day; the question decides.

resample() returns a Resampler until you call an aggregation such as sum(), mean(), count() or agg(). Use on="date" when the datetimes are in a column instead of the index.

Read calendar bins and partial coverage

A daily series of ones from January 1 through April 30, 2025 gives monthly totals 31, 28, 31 and 30 with resample("ME").sum(). "MS" gives the same monthly totals with start labels. Quarter-end resampling gives 90 for January–March and 30 for a second quarter containing April only. A June 30 label does not establish that May and June were observed.

Weekly "W" bins end on Sunday. Daily values 1 through 14 starting Monday, March 3 sum to 28 for the week labeled March 9 and 77 for the week labeled March 16. "W-MON" ends on Monday; the first bin then contains March 3 alone. Read the weekday anchor before treating a label as a week's start.

Separate membership from labeling

closed controls which boundary belongs to a bin. label controls the timestamp printed on the result. For most rules, including "5min", "h", "D" and "MS", both default to "left".

Values 0 through 9, one per minute from 00:00, produce two default five-minute sums: 10 labeled 00:00 and 35 labeled 00:05. The first bin contains 00:00 through 00:04. With closed="right", label="right", results are 0 at 00:00, 15 at 00:05 and 30 at 00:10. The middle bin contains 00:01 through 00:05. Changing only the label would not change membership.

End rules such as "ME", "QE", "YE", "W" and "BME" default to right closure and right labels. Fixed-interval bins normally align from midnight of the first day, so 15-minute boundaries are 00:00, 00:15 and so on.

Distinguish empty bins from real zeros

An empty day's sum() is 0, count() is 0 and mean() is NaN. Use sum(min_count=1) to preserve missingness when no reading is available. A recorded zero and an absent reading then remain distinguishable.

For several summaries of one series, .ohlc() gives open, high, low and close: the first, maximum, minimum and last observed value in each bin. Readings 10, 12 and 11 give open 10, high 12, low 10 and close 11. Open and close depend on time order, not on numerical size.

A finer grid creates slots, not measurements. Before filling them, decide whether each value represents a state, a count, a smooth measurement or a total over a span.

Expose the new slots

s = pd.Series([10.0, 16.0],
              index=pd.to_datetime(["2025-06-01", "2025-06-04"]))
daily = s.asfreq("D")

The daily result is 10, NaN, NaN, 16. s.resample("D").asfreq() gives the same grid here. Values are kept at matching timestamps; the new dates contain no original observation.

Fill according to meaning

A thermostat setting holds until someone changes it, so forward fill is useful. A smoothly changing water level can justify interpolation. Zero suits an event count only when the missing record means no event. A sensor outage does not establish a zero measurement.

The interpolation example uses a regular daily grid. The total change is 6 across three one-day steps, so each step adds 2. The two interior values are estimates, not newly recovered measurements.

Backward fill uses future information. It cannot describe what an observer knew at each earlier moment. Forward fill respects that time direction, but long gaps can still make a carried state stale. daily.ffill(limit=1) fills one slot after a reading, giving 10, 10, NaN, 16 in this example.

Use explicit reporting boundaries

asfreq("D") spans the first through last source timestamp. Reindex when the calendar must extend further:

calendar = pd.date_range("2025-06-01", "2025-06-05", freq="D")
complete = s.reindex(calendar)

June 5 is now visible too. fill_value=0 in reindexing fills newly created labels, which is useful for verified zero-event days. It does not replace missing values that already existed in the source. Both reindexing and asfreq() need unique source labels.

Keep sampling separate from aggregation

asfreq() selects values at target timestamps. It does not summarize the observations between them. On 48 hourly values numbered 0 through 47, asfreq("D") selects midnight readings 0 and 24. resample("D").sum() instead gives 276 and 852 by adding every hourly value in each day.

A monthly total cannot be forward-filled onto every day and then summed: that would count the same total repeatedly. An equal daily allocation requires dividing by the month's number of days. That is an allocation assumption, distinct from carrying a state or estimating a smooth measurement.

A long table can hold many series in one value column. A store key distinguishes stores, and a sensor key distinguishes sensors. Group by that identity as well as time when the question asks for a separate history for each series.

Combine a series key with a time rule

monthly = df.groupby([
    "store", pd.Grouper(key="date", freq="ME")
])["sales"].sum()

pd.Grouper places a time-bin rule inside groupby. key= names a datetime column; level= can name a datetime index level. The result has one index level for store and another for month end. Selecting the numeric column keeps the aggregation focused on sales.

Suppose store A sells 100 on January 5, 150 on January 20 and 120 on February 3. Store B sells 80 on January 11, 90 on February 14 and 60 on February 25. The grouped result is A: January 250, February 120; B: January 80, February 150. Resampling the entire table without the store key would combine the stores into January 330 and February 270.

Resample within each group

A second form makes the date the index first:

monthly = (df.set_index("date")
             .groupby("store")["sales"]
             .resample("ME").sum())

It produces the same totals for observed store-month combinations. The two forms differ when a store has no rows in an intervening month. If A appears in January and March only, group-wise resampling creates a February bin between A's first and last date; ordinary sum() gives zero there. Grouping by store and a Grouper retains the combinations that contain rows, so A's February combination is absent.

An absent combination, an empty bin summarized as zero and a missing measurement are different data states. Decide which one the report should show. Use explicit reindexing when every store must share a calendar wider than its observed range. Use min_count=1 with a sum when an empty bin should stay missing.

Reshape for comparison

monthly.unstack("store") moves the store level into columns and leaves months in rows. In the example, January has A = 250 and B = 80; February has A = 120 and B = 150. A missing store-month combination becomes NaN; unstack("store", fill_value=0) replaces missing combinations with zero when that meaning is appropriate.

Keep years separate when needed

A time Grouper at "ME" keeps January 2024 and January 2025 separate. Grouping by df["date"].dt.month keeps only month numbers and merges both Januaries. The first form answers "What happened each month?" The second supports a seasonal comparison such as average January sales across years. The choice is about the time identity you need to preserve.

A rolling window repeatedly summarizes a moving group of rows. At each new row, the window gains the current value and eventually drops its oldest one. The window length controls how much short-term variation is smoothed.

Work one window at a time

For values 4, 8, 6, 10 and 12:

s = pd.Series([4, 8, 6, 10, 12])
s.rolling(3).mean()

The first full window is [4, 8, 6], totaling 18 and averaging 6. The next is [8, 6, 10], averaging 8. The last is [6, 10, 12], averaging about 9.333. Default results are NaN, NaN, 6, 8 and 9.333 because the first two rows do not have three observations yet.

Control data requirements and placement

min_periods sets the required count of non-missing values. For a row-count window it defaults to the full window size. With min_periods=1, the early means use one value and then two: 4 and 6. Their smaller samples make them less stable than the later three-value means.

Missing values still occupy row positions. In [4, NaN, 6, 10, 12], a window of three rows containing the missing value has only two valid observations. min_periods=2 permits the mean of [4, NaN, 6], which is 5; the default does not.

Results normally belong to the window's final row. With center=True, the same three full-window means appear at the middle rows, giving NaN, 6, 8, 9.333, NaN. A centered value uses later observations relative to its label. Use trailing windows to describe what was known at each moment.

Summarize more than the mean

The same windows support sum(), max(), min() and std(). pandas uses sample standard deviation by default: [4, 8, 6] has standard deviation 2. A custom function can compute a range:

s.rolling(3).apply(lambda v: v.max() - v.min(), raw=True)

The full-window ranges are 4, 4 and 6. raw=True passes array values to the function. Two aligned series can also share a window: a.rolling(3).corr(b) measures correlation across the last three paired observations, so the relationship can vary over time.

Choose a meaningful length and grouping

With one complete daily row per day, a seven-row average contains every weekday once and averages out a fixed repeating weekday component. A twelve-row average does the same for a fixed annual component on complete monthly data. Longer windows smooth more but respond later to turns; a trailing n-row average has roughly half a window of delay, about (n − 1) / 2 rows.

Rows are not days. Seven readings with gaps can cover more than a week. Use a time window when the question names elapsed time. In a long table, sort each series in time and run windows within its group: df.groupby("store")["sales"].rolling(2).sum(). That prevents a window from crossing from one store's data into another's.

Irregular events call for windows defined by time. "Orders in the past three days" should keep the same duration even when there are many orders one day and none the next.

Define an elapsed-time window

s = pd.Series([5, 3, 4, 6], index=pd.to_datetime([
    "2025-05-01", "2025-05-02", "2025-05-05", "2025-05-06"
]))
s.rolling("3D").sum()

The results are 5, 8, 4 and 10. Unlike rolling(3), the string "3D" defines a span of three days. The number of rows inside that span can vary. Sort the DatetimeIndex first; time windows require monotonic times. A DataFrame can use a sorted datetime column with on="date".

Test the boundary before adding

By default the window includes the current time and excludes the instant exactly three days earlier. At May 5 midnight, the interval is after May 2 midnight through May 5 midnight. The May 2 reading is exactly on the excluded boundary, so only May 5's value of 4 contributes.

At May 6, the window begins just after May 3 and includes values 4 and 6, totaling 10. With closed="both", May 5's window also includes May 2 and totals 7. A boundary choice can change a result even when the data and duration stay the same.

A two-row window on this series gives NaN, 8, 7 and 10. At May 5 it includes the previous row from May 2. That answers "the last two readings", a different question from "the last three days".

Set an evidence requirement

A time window's min_periods defaults to 1. The first observation can therefore produce a result immediately. s.rolling("3D").count() gives 1, 2, 1 and 2 readings, exposing the changing support for the result. A mean based on one reading and a mean based on many are both valid calculations but have different data support.

With min_periods=2, the sums are NaN, 8, NaN, 10. The duration has not changed; the calculation now requires at least two non-missing observations inside that duration. A wider time interval does not itself guarantee more observations.

Use fixed lengths and separate series

Time-based rolling requires a fixed duration such as "30min", "24h" or "7D". A month-end rule such as "1ME" has variable length and cannot define this kind of rolling window. Use a stated fixed length such as "30D" if that is the intended rule, or aggregate into monthly bins first when the question concerns calendar months.

For separate customer histories, df.groupby("customer").rolling("30D", on="date")["amount"].sum() keeps each customer's transactions together. Times must be ordered within each customer. The result adds the group key as an index level, preserving which series each window describes.

Some questions need the whole history: "How much has accumulated?" or "What is the highest value so far?" Others should favor recent observations without abruptly discarding the past. Expanding and exponentially weighted calculations serve these different aims.

Grow the history one row at a time

For values 4, 8, 6, 10 and 12, s.expanding().mean() gives 4, 6, 6, 7 and 8. The fourth result uses all four values: their total is 28 and their mean is 7. expanding().sum() gives 4, 12, 18, 28 and 40, matching cumsum() on this complete series. expanding().max() gives 4, 8, 8, 10 and 12, matching cummax().

Expanding windows default to one required observation. They keep every earlier observation with equal weight. Late in a long series, a new value contributes only a small part of that entire history, so a running mean reacts slowly.

Running totals can restart at a boundary:

ytd = df.groupby(df.index.year)["sales"].cumsum()
by_store = df.groupby("store")["sales"].cumsum()

Sort within the relevant history before accumulating. A plain cumulative sum over interleaved stores mixes them.

Measure distance below a running peak

For 100, 120, 90, 110 and 130, cummax() gives 100, 120, 120, 120 and 130. Compute drawdown with s / s.cummax() - 1. At 90 the reference is the earlier peak of 120, so the drawdown is −0.25, or 25% below the peak. Comparing 90 with the first value of 100 would answer a different question. A new peak has drawdown zero.

Give recent values more weight

An exponentially weighted moving average, or EWMA, reduces the weight of older values by a constant factor. The smoothing factor follows $\alpha =2\mathrm{/}(\text{span}+1)$, so span=3 gives 0.5. For adjust=False, the update is:

Here, $x_t$ is the new observation, $y_{t-1}$ is the previous smoothed value and $y_t$ is the updated result. The first smoothed value equals the first observation. Then each new value receives weight 0.5 and the previous smoothed value receives weight 0.5:

s.ewm(span=3, adjust=False).mean()

For 4, 8, 6, 10 and 12, the results are 4, 6, 6, 8 and 10. The final update averages the new observation 12 and the previous smoothed value 8. It does not average only the two most recent raw observations.

Read the adjustment setting

The default adjust=True normalizes by the sum of available weights. For the same data, the early results are 4, about 6.667, 6.286, 8.267 and 10.194. This is a different startup calculation from the recursive form; the difference fades as more history becomes available.

Choose one decay argument: span, alpha, com or halflife. An EWMA has no hard cutoff where an old row suddenly disappears. A rolling mean has that cutoff, while an expanding mean keeps all earlier values equally weighted. Choose according to how much influence recent changes should have.

A clock reading needs a location to identify a moment. "2025-07-01 09:00" is a naive timestamp. Adding "Europe/Amsterdam" makes it aware: in July it has offset +02:00 and represents 07:00 UTC. UTC is the shared time reference and has no daylight saving clock changes.

Attach the recording zone

local = pd.Timestamp("2025-07-01 09:00")
aware = local.tz_localize("Europe/Amsterdam")

tz_localize() keeps 09:00 and assigns its zone meaning. Use the location where the clock reading was recorded. Calling the same reading UTC would describe a different moment, two hours later in this example.

Use IANA names such as "Europe/Amsterdam", "America/New_York" and "Asia/Kolkata". Named regional zones retain historical offset and daylight saving rules. A fixed +01:00 offset does not. Short names are inconsistent: "EST" is fixed at −05:00, while "CET" follows seasonal changes. pandas 3 uses Python's zoneinfo representation.

Convert an already identified moment

aware.tz_convert("UTC") gives 07:00 UTC. Converting that same instant to New York gives 03:00, to Kolkata 12:30, to Tokyo 16:00 and to Sydney 17:00 on July 1. The clock changes, but elapsed-time identity stays the same.

Converting a naive value raises an error because no starting zone is known. Localizing an aware value also raises an error; use conversion for that operation. On columns, use .dt.tz_localize() and .dt.tz_convert(). On indexes use the methods directly, or the DataFrame's index-oriented methods.

Read offset-bearing text and combine zones

A trailing "Z" in ISO text means UTC. For text carrying different offsets, parse with utc=True:

raw = pd.Series(["2025-03-01 10:00+01:00",
                 "2025-04-01 10:00+02:00"])
common = pd.to_datetime(raw, utc=True)

These become 09:00 and 08:00 UTC. Convert afterward to a regional zone for local reporting. Mixed-offset parsing without utc=True raises in pandas 3.

Two aware series align by actual moments even when their zone labels differ. Amsterdam readings at 09:00 and 10:00 and London readings at 08:00 and 09:00 in July refer to matching instants. Their arithmetic result uses UTC labels. Ordering comparisons between naive and aware timestamps fail; equality returns false.

Remove a zone deliberately

tz_convert(None) first expresses an aware value in UTC and then removes the zone. Amsterdam's 09:00 becomes naive 07:00 in July. tz_localize(None) removes the zone while keeping naive 09:00. These are different outputs. Keep calculation data aware in UTC, then convert before reporting or grouping by a local day.

Local clocks can skip or repeat an hour. Those changes affect parsing, elapsed durations and daily totals. Named time zones let pandas distinguish the actual moments behind local clock readings.

Recognize a skipped local time

Amsterdam advanced from 02:00 to 03:00 on March 30, 2025. The clock reading 02:30 never occurred. Localizing it without a policy raises an error:

pd.Timestamp("2025-03-30 02:30").tz_localize(
    "Europe/Amsterdam", nonexistent="shift_forward"
)

The result is 03:00+02:00, the first valid time after the gap. nonexistent="NaT" preserves the problem as missing. "shift_backward" moves to the last valid moment before the gap. Shifting a record changes its timestamp, so retain the original input and document the chosen policy in the analysis.

Distinguish two repeated readings

On October 26, 2025 Amsterdam moved from 03:00 back to 02:00. The local reading 02:30 occurred once with offset +02:00 and once with +01:00. They are an hour apart in UTC.

ambiguous=True selects the daylight saving occurrence; ambiguous=False selects standard time. ambiguous="NaT" marks an unresolved reading as missing. A boolean list can choose per row. ambiguous="infer" can use an ordered sequence containing the repeated hour, such as 01:30, 02:00, 02:30, 02:00, 02:30, 03:00. A lone repeated clock reading does not provide enough ordering evidence to infer which occurrence was intended.

Count hours in the local day

The spring transition day has 23 hours and the autumn transition day has 25. A complete hourly series resampled to local days therefore has varying counts. To report average hourly energy, divide a daily sum by the actual number of valid hourly readings, or take their mean. Dividing every total by 24 misstates the transition days.

A partial dataset can also create short bins. A UTC series starting March 29 at 00:00 begins in Amsterdam at 01:00, so its first local day lacks midnight even before the transition day. Check the reporting boundaries as well as the zone rules.

Distinguish same clock time from equal elapsed time

t = pd.Timestamp("2025-03-29 12:00", tz="Europe/Amsterdam")
t + pd.offsets.Day(1)       # March 30 at 12:00
t + pd.Timedelta(days=1)    # March 30 at 13:00

The calendar offset preserves noon and crosses only 23 elapsed hours. The Timedelta adds 24 hours. In autumn, a local calendar day can instead contain 25 elapsed hours.

Choose the reporting zone before grouping

Resampling follows the index's zone. To total use by Amsterdam's local date, first convert an aware UTC index to "Europe/Amsterdam", then resample to "D". Resampling in UTC first creates different day boundaries that cannot be repaired merely by relabeling the result.

Transition dates differ by region: New York changed on March 9 and November 2 in 2025; Sydney ended summer time on April 6 and began it on October 5. Tokyo and Kolkata have no daylight saving changes. Keep raw moments in UTC and use the named local zone for calendar reports.