Exploratory Data Analysis in Python

Exploratory data analysis, or EDA, uses summaries and plots to learn what a dataset contains. You look for quality problems, patterns and surprises, then turn them into better questions. John Tukey developed and promoted this approach, most notably in his 1977 book Exploratory Data Analysis.

Start with a decision

A request such as "How are our deliveries doing?" is too broad to guide the first calculation. Ask what decision the analysis will support, then name the variables and comparisons that matter. Useful questions include "What is the typical delivery time, and how much does it vary?", "What share of orders takes longer than 45 minutes?", "Do times differ by city or hour?" and "Has the monthly median changed since January?" Each points toward a summary, a plot or a comparison.

Before computing, establish where the data came from, how they were collected, their period and population, and each column's meaning and unit. A jump in completed fields may reflect a form change that made a field required. That context changes the explanation of the pattern.

Know what one row means

The unit of observation, also called granularity, is the thing represented by one row. It might be an order, an item within an order, a customer or a store on a particular day. A table with 12,000 order-line rows and 4,000 distinct order IDs averages 3 rows per order. Its row count describes items, so an order-level question needs an order-level count. Repeated order IDs are expected in that table; they would be a quality problem in a table promised to contain one row per order.

Work through a loop

A useful workflow is:

1. State the questions and the decision they support.
2. Load the data and confirm the meaning of a row.
3. Profile structure, types and quality.
4. Examine important variables individually.
5. Explore relationships between variables.
6. Compare segments and periods.
7. Write findings, limitations and next questions.

These steps form a loop. A surprising relationship can send you back to the source to check units or missing values. New data can change the profile and reopen earlier questions. Starting with questions keeps the analysis focused; staying open to surprises keeps it exploratory. Plotting every possible pair without a purpose usually creates a long chart collection with little to explain.

Distinguish a lead from confirmation

Exploration generates hypotheses. Confirmatory analysis checks a hypothesis stated in advance, ideally using data that did not reveal the pattern. Looking through many charts makes chance patterns more likely, so a striking exploratory result needs follow-up before it supports a general claim. EDA helps prepare that work by showing usable variables, skewed distributions and promising relationships.

The data's coverage also sets the scope of a finding. Product reviews come from people who chose to write reviews. Their ratings describe those respondents; extending the result to all customers requires evidence about how respondents differ from everyone else.

A Jupyter notebook combines code, outputs and Markdown explanations. Its kernel is the Python process that holds variables in memory. The kernel remembers the order in which cells ran, which can differ from the order visible on the page.

Remove hidden state

Suppose you run a cell that creates clean, delete that cell, and continue using clean below. The current session still works because the variable remains in memory. A fresh session fails because the notebook no longer contains its definition. Editing an early cell after running later calculations can also leave old outputs beside new code.

Execution counts such as [7] help reveal out-of-order runs. The practical test is stronger: in JupyterLab, choose Kernel → Restart Kernel and Run All Cells. Check that the whole notebook completes and its numbers agree with the outputs you intend to share.

Put dependencies in a clear order

Start with a title, purpose and questions. Follow with imports and a settings cell, then sections for loading, quality checks, distributions, relationships, comparisons and findings. For example:

from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

DATA = Path("data/raw/orders.csv")
LATE_MINUTES = 45
SEED = 42

df = pd.read_csv(DATA)

A relative path works inside the shared project structure. A path tied to one person's desktop breaks when the project moves. State the working directory used to run the notebook so the relative path resolves correctly. Preserve raw files and save processed data separately. Make changes in code: a manual spreadsheet edit is invisible to the next rerun.

Record the environment and random choices

Save package versions with an environment file, such as the output of pip freeze > requirements.txt, and print important versions in the notebook with print(pd.__version__). Behavior changes across releases: pandas 3 gives text a dedicated str dtype, and a DataFrame containing text needs an explicit numeric selection for correlation.

A sampled plot should show the same observations on each rerun. Use df.sample(n=1000, random_state=42) when at least 1,000 rows are available, or create a NumPy generator with rng = np.random.default_rng(42). Record the seed with the other settings. A seed controls random choices; reproducing the complete output also requires the same inputs and software.

Make assumptions executable

Turn important expectations into checks:

assert df["order_id"].is_unique
assert df["minutes"].ge(0).all()

The first requires one unique order ID per row. The second requires every duration to be nonnegative and present for ordinary floating-point data. If missing times are allowed, define a separate missingness check and apply the range rule only to recorded times. A failed assertion stops the run where the input first violates the expected contract.

Put repeated procedures into functions so one correction updates every use. Keep code and notebooks under version control. Notebook outputs are stored inside their JSON, which can make changes hard to compare; clearing outputs before a commit or pairing the notebook with a text script through Jupytext can help.

A first inspection establishes the table's size, what its columns represent and whether their stored types suit those meanings. This profile helps you choose sensible summaries before trusting the output of a convenient command.

Inspect structure from several angles

Use df.shape for the row and column counts, df.head() and df.tail() for the ends of the file, and df.sample(5, random_state=0) for a reproducible sample when at least five rows exist. The first rows alone may all belong to the earliest date or the same store. Run df.info() to inspect column names, non-null counts, dtypes and memory use. Use df.nunique() to count distinct recorded values in each column.

Separate meaning from storage

An integer dtype does not make every column a quantity. Region codes and customer IDs have numeric storage but no useful arithmetic mean. A rating from 1 to 5 has a small set of ordered responses and often benefits from a categorical display.

pandas commonly uses int64, float64, bool, datetime64, category and, from pandas 3, str. Earlier pandas versions often used object for text. Ordinary integer data may become float64 when missing values occur because NaN is a floating-point marker; nullable Int64 supports integers with missing values.

Check conversions and keys

A numeric-looking string can hide separators, units or missing codes. For a Series containing "1,234", "950", "n/a" and "1,010":

parsed = pd.to_numeric(
    s.str.replace(",", ""), errors="coerce"
)

The values become 1234.0, 950.0, NaN and 1010.0. Compare missing counts before and after conversion, and inspect newly missing entries. Coercion also turns an unhandled value such as "12 kg" into NaN, so successful execution is not enough to show that the conversion preserved the data.

Read codes with meaningful leading zeros as text, for example pd.read_csv(path, dtype={"postal_code": "str"}). Recovering a lost zero later requires knowledge that the integer representation discarded.

Identify the expected row key. df["order_id"].is_unique checks a single-column key; df.duplicated(subset=["store_id", "date"]).sum() checks repeated store-date combinations. Relate key counts back to the row meaning. A single-valued column cannot explain differences within this dataset; thousands of supposed city categories may signal inconsistent labels.

Complete a data dictionary as you inspect: record each column's meaning, analysis type, unit, allowed values and known issues. Those entries supply the rules for the next quality checks.

Missing values change which observations contribute to a result. Duplicate records can give some observations extra weight. Measure both before choosing a treatment, and keep the decision visible in the analysis.

Measure missingness and its pattern

pandas uses NaN, NaT for missing datetimes and <NA> for nullable types. Count and rank missingness with:

missing_count = df.isna().sum()
missing_share = df.isna().mean().sort_values(ascending=False)
rows_with_gaps = df.isna().any(axis=1).sum()

A Boolean mean gives a share because True counts as 1 and False as 0. With 96 missing ages among 1,200 rows, the missing share is 8%. Rows with any gap can be much more numerous than missing entries in one column; this is the row loss that a blanket df.dropna() would cause.

Some gaps hide behind ordinary values. Default CSV parsing recognizes many empty and text missing markers, but codes such as -999, "unknown" and "?" may survive. Inspect frequencies and extremes, check the data dictionary, and declare additional markers with na_values. Zero needs context: it can mean no returns, or an undocumented placeholder.

Check whether gaps concentrate in a segment:

age_gaps = df["customer_age"].isna()
age_gaps.groupby(df["city"]).mean()

If one city has 40% missing ages and the others about 2%, complete-row analysis will underrepresent that city. The terms MCAR, MAR and MNAR describe different mechanisms: missingness unrelated to observed or missing values; missingness depending only on observed variables; and missingness depending on the missing value itself. High-income respondents skipping an income question illustrate MNAR. EDA can detect links with recorded variables; deciding the full mechanism needs more evidence about collection.

Name the denominator

For 6 card payments, 2 cash payments and 2 gaps, value_counts(normalize=True) reports 75% card and 25% cash among the 8 recorded payments. Adding dropna=False reports 60% card, 20% cash and 20% missing among all 10 rows. Neither table answers the other's question. Label the population behind each share.

Inspect full-row and key duplicates

df.duplicated().sum() counts copies after the first occurrence of each full row. A key-based check can find conflicting records that are not identical:

repeated = df["order_id"].duplicated(keep=False)
df.loc[repeated].sort_values("order_id")

Consider IDs 101, 102, 102, 103, 104, 104 with amounts 20, 35, 35, 12, 50, 55. There is 1 full-row duplicate, 2 repeated key occurrences after the first, and 4 rows involved in repeated keys. The two 104 records disagree about the amount, so deleting a copy without investigation could discard the correct value.

Record the count, likely cause, treatment and affected rows for each issue. Dropping all gaps can remove a biased subset; filling them with the mean hides their pattern and reduces apparent spread. Choose a treatment for the question and collection process, then keep the original missingness measurable.

A value is valid when it meets its column's allowed range or set. Records are consistent when their values agree across columns, rows and sources. Build checks from the data dictionary and knowledge of the process that created the data.

Check ranges without mixing in gaps

Start with minima and maxima, then test explicit limits. To count recorded ages outside 0–120:

invalid_age = df["age"].notna() & ~df["age"].between(0, 120)
invalid_age.sum()

between returns False for missing values. Negating it alone would count missing ages as invalid ages; the presence check separates the issues. Other rules might require nonnegative quantities, percentages no greater than 100, or extraction dates no earlier than the events they contain. The permitted range comes from the column's meaning.

Reconcile labels and columns

"Lyon", "lyon" and " Lyon" are three strings, even when they name the same city. Counts of 410, 22 and 5 become 437 for "lyon" after a justified trim and case conversion:

city_clean = df["city"].str.strip().str.lower()

Compare category counts before and after. Spelling variants and synonyms need an explicit mapping; changing case alone cannot merge "UK" with "United Kingdom". Retain the rule used so a rerun makes the same decision.

Cross-column checks look for contradictions. df.query("delivered_at < ordered_at") finds deliveries timestamped before their orders. Other checks ask whether item amounts add to the total, age agrees with birth date, or a canceled order has an unexpected delivery time. These rules require shared units, time conventions and definitions across columns.

Look for scale and coverage changes

Mixed kilograms and grams can create clusters roughly 1,000 times apart. A share recorded partly on a 0–1 scale and partly on a 0–100 scale also creates a misleading distribution. Check the source before converting: two clusters can also represent two genuine populations. A plot suggests where to investigate; the source establishes which units apply.

Inspect the earliest and latest dates and counts by month. A missing month may be an extraction gap, while a partial last month naturally contains fewer records. An abrupt increase can reflect a new recording system. Record the coverage you actually have before reporting a time pattern.

Keep a quality log

Use a small table with the issue, check, number of affected rows, evidence and action. For example, a negative duration can be flagged by a nonnegative-time rule, investigated against timestamps and corrected only when the source establishes its true value. Important expectations can become assertions that stop a future run.

A 120-minute delivery is unusual but possible. A −5-minute duration is impossible under the same duration definition. The first needs an outlier investigation; the second needs an error investigation.

A numeric distribution describes which values occur and how often. Its center gives a typical location, its spread shows variation, and its shape reveals what those summaries leave out. Use a plot alongside the numbers.

Choose summaries that fit the distribution

The mean is the sum divided by the count. The median is the middle sorted value, or the mean of the two middle values for an even count. The range is maximum minus minimum. The interquartile range, or IQR, is Q3 minus Q1: the distance between the 75th and 25th percentiles. Standard deviation measures spread around the mean in the variable's own units.

pandas s.std() uses the sample calculation with ddof=1, dividing the squared-deviation total by one less than the count. NumPy np.std(a) defaults to ddof=0, dividing by the count. This matters especially with small samples, so use matching settings when comparing results.

Read the summary before interpreting it

s = df["minutes"]
s.describe(percentiles=[0.1, 0.5, 0.9])

The result includes the non-missing count, mean, standard deviation, minimum, requested percentiles and maximum. The 50% entry is the median. On a DataFrame, describe() covers numeric and datetime columns by default; datetime summaries include their mean and percentiles. describe(include="all") also summarizes other columns. Text and categorical columns receive count, unique, top and freq, where top is a most frequent value and freq is its count. Exclude numeric identifiers from arithmetic summaries.

Work through a long right tail

Consider delivery times of 18, 22, 25, 25, 27, 29, 31, 34, 38, 41 and 95 minutes. Their sum is 385 and their count is 11, so the mean is 35.0 minutes. The sixth sorted value is 29, giving the median. pandas gives Q1 = 25, Q3 = 36, IQR = 11 and a sample standard deviation of about 21.02 minutes.

Without the 95-minute delivery, the mean is 29.0, the median 28 and the standard deviation about 7.15. One value moves the mean by 6 minutes and roughly triples the standard deviation; the median moves by 1 minute. This is why the median and IQR are useful resistant summaries for data with extreme values. A roughly symmetric distribution without extremes is often well described by its mean and standard deviation.

Describe the tail and useful positions

Right skew has a long high-value tail; left skew has a long low-value tail. The mean usually lies toward the long tail relative to the median, but check the shape because this rule of thumb has exceptions. s.skew() is positive for a right tail and negative for a left tail; it is about 2.71 for these delivery times.

s.quantile(0.9) gives 41 minutes here, a useful description of the upper part of the distribution. One of the 11 deliveries exceeds 45 minutes, about 9.1%. Compute threshold shares over recorded values when gaps exist. Small-sample quartiles can differ between programs: pandas uses linear interpolation by default. Record the method when exact quartiles matter.

A distribution plot reveals tails, peaks, gaps and limits that a center alone hides. Each plot emphasizes different features, so choose it for the question and check how display choices affect the picture.

Use meaningful histogram bins

A histogram divides a numeric scale into intervals and counts the observations in each. sns.histplot(data=df, x="minutes", binwidth=5) uses five-minute bins. seaborn chooses automatic bins by default; pandas and Matplotlib histograms default to 10. Try several reasonable bin settings. A very broad bin can merge two peaks, while very narrow bins can turn sampling noise into apparent structure. For integer counts, discrete=True centers a bar on each integer.

Describe the shape as symmetric, right- or left-skewed, uniform, or having one, two or several peaks. Look for gaps, caps and floors. Spikes at round values such as 30 and 60 minutes can reflect heaping from rounded self-reports. Two peaks near 25 and 50 minutes might reflect bike and car deliveries. Adding hue="vehicle" checks whether those groups explain the split.

Compare the main plot choices

A kernel density estimate, or KDE, smooths observations into a curve. sns.kdeplot(data=df, x="minutes") can be helpful, but bandwidth changes its shape. Smoothing can conceal caps or extend below zero even when durations are nonnegative. Read that extension as a feature of the estimate, not as negative observed durations.

A box plot draws the median inside a box from Q1 to Q3. Default whiskers reach the most extreme observed values still within 1.5 IQR of the box; points farther away appear separately. A whisker is therefore not necessarily the minimum or maximum, and its end is an observed value rather than the calculated fence.

Read cumulative shares and multiplicative scales

sns.ecdfplot(data=df, x="minutes")

An ECDF shows the share less than or equal to each value. Read horizontally from a share such as 0.9 to locate a percentile. It avoids the bin choice of a histogram and is useful for comparing how often values stay below a threshold.

A log scale helps with positive data spanning large ratios. On a base-10 scale, 10, 100 and 1,000 are equally spaced because each step multiplies by 10. Ordinary real logarithms require positive values; np.log1p(x) computes the natural logarithm of one plus the value and handles zero in nonnegative data.

When comparing differently sized groups, normalize within each group using stat="percent", common_norm=False, or use panels with shared axes. Finish with a numerical description, such as "Delivery times are right-skewed, with a median of 29 minutes and half of the orders between 25 and 36 minutes; one order took 95 minutes."

Categorical summaries count observations in named groups. Their meaning depends on the labels, their order and the population used for each percentage. A numeric category code still needs a category summary.

Build a frequency table

s.value_counts() lists categories from most to least frequent. Use normalize=True for proportions and dropna=False to include a missing-value row. For text, describe() reports the number recorded, the number of distinct values, a most frequent value and its count.

In 200 survey responses, 30 ratings are "low", 50 are "medium" and 120 are "high". Dividing each count by 200 gives 15%, 25% and 60%. If those are the 200 people who answered a question in a larger survey, the percentages describe respondents to that question. Missing responses change the denominator when they are included.

Preserve a meaningful order

Nominal categories such as cities have no inherent rank; sorting by frequency can make comparisons easy. Ordinal categories have a natural order that should stay visible:

order = ["low", "medium", "high"]
counts = s.value_counts().reindex(order, fill_value=0)

An ordered categorical records the order for sorting, comparisons and plots. Its value_counts() still sorts by frequency by default; use sort=False or explicitly reindex the counts. A chart ordered "high", "medium", "low" because of frequency can make the scale harder to recognize.

Keep rare labels and gaps distinct

When many categories make a plot unreadable, retain the most common labels and collect the rest as "Other":

top = s.value_counts().nlargest(5).index
grouped = s.where(s.isna() | s.isin(top), "Other")

The presence of s.isna() keeps gaps missing. Without it, missing entries would also become "Other", mixing nonresponses with real rare answers. Report how many labels and observations were combined. Check that a rare category important to the question has not disappeared into the combined group.

Horizontal bar charts suit long labels. For example, sns.countplot(data=df, y="city", order=df["city"].value_counts().index) sorts city counts. Start the count axis at zero because bar length encodes the number of observations. A pie chart becomes difficult to compare when it contains many slices.

Turn numeric ranges into labeled categories

bands = pd.cut(
    df["minutes"], bins=[0, 30, 45, np.inf],
    labels=["up to 30", "over 30 to 45", "over 45"],
    include_lowest=True
)

The default right-closed intervals put 30 in the first band and 45 in the second. include_lowest=True also includes 0 in the first band. Values outside these ranges become missing in the binned result, so validate negative durations before interpreting its missing count. Record the boundaries because a label such as "30–45" alone leaves the treatment of 30 unclear.

For nominal codes, map 1 and 2 to their region labels before displaying them. Averaging region codes would describe their arbitrary numbering, not the distribution of regions.

An outlier lies far from the bulk of the data under a stated rule and context. It may be a recording error, a genuine rare event or a member of a separate group. A flag starts an investigation.

Calculate IQR fences

The IQR rule flags values below or above these limits:

For delivery times 18, 22, 25, 25, 27, 29, 31, 34, 38, 41 and 95, Q1 is 25 and Q3 is 36. The IQR is 11, so the lower fence is 8.5 and the upper fence is 52.5. The 95-minute order is flagged.

q1, q3 = s.quantile([0.25, 0.75])
iqr = q3 - q1
flag = (s < q1 - 1.5 * iqr) | (s > q3 + 1.5 * iqr)

These are the default fences underlying box-plot outlier points. A value exactly on a fence is inside it. In a long right tail the rule can flag many genuine high values.

Understand masking and robust scores

A z-score measures distance from the mean in standard-deviation units:

A common flag is an absolute z-score greater than 3. Here the mean is 35 and the sample standard deviation is about 21.02, so 95 gives a z-score of about 2.85. The extreme value has increased the very center and spread used to judge it. This effect, called masking, can let an unusual value escape the rule. The rule also fits roughly symmetric distributions better than skewed ones.

The median absolute deviation, or MAD, is the median of the absolute distances from the median. It uses resistant quantities. A modified z-score is:

Here the median is 29 and MAD is 5. For 95 minutes, the modified score is about 8.90, exceeding the common absolute threshold of 3.5. If MAD is zero, this division is undefined; inspect the repeated values and use another justified check instead of reporting an infinite score as a measured result.

Investigate before changing data

Domain rules come first. A negative elapsed duration is an error under this definition regardless of its statistical score. For a possible extreme, inspect the whole row, its timestamps, distance, group and source. A storm or a long route may explain a slow delivery.

Comparing results with and without the observation is a sensitivity check. Capping extremes at a selected percentile, called winsorizing, is another deliberate transformation; report the cap and that values changed. Also inspect unusual combinations: two individually plausible measurements can form a rare point in a scatter plot.

A scatter plot places one observation at each pair of numeric values. Put a possible explanatory variable on the horizontal axis and the response on the vertical axis. The plot helps you see the form of a relationship before compressing it into a coefficient.

Describe four features

Use direction, form, strength and unusual points. Positive direction means the response tends to rise as the horizontal variable rises; negative means it tends to fall. Form might be straight, curved or split into clusters. Strength describes how closely points follow that form, which can be strong even when the form is curved.

For temperatures of 8, 12, 15, 18, 21, 24, 27 and 30 °C, daily bike rentals are 310, 420, 515, 600, 690, 760, 800 and 780. The relationship is strong and positive overall. It is close to linear up to about 27 °C, then levels off and falls slightly. Describing it only as "positive" misses the change at the warm end.

sns.scatterplot(data=df, x="temp_c", y="rentals")

Make dense plots readable

With many rows, points cover one another: overplotting hides where observations concentrate. Smaller markers and alpha=0.2 reveal some density through transparency. A hexbin plot aggregates observations into hexagons:

df.plot.hexbin(x="distance_km", y="minutes", gridsize=30)

A two-dimensional sns.histplot provides a similar view with rectangular bins. A random sample can reduce clutter; fix its seed so reruns show the same subset. When an axis takes only a few values, jitter spreads stacked points for visibility. The jitter is display movement, not extra measured precision.

Add groups and check a fitted line

Use hue="season" for color or sns.relplot(..., col="season") for separate panels. A pooled cloud may contain distinct seasonal clusters. Marker size can encode another quantity when its meaning stays clear.

sns.regplot(data=df, x="temp_c", y="rentals") adds a least-squares straight line and a shaded uncertainty band. A line helps the eye but can miss a bend. lowess=True draws a smooth curve when the statsmodels dependency is installed. Inspect the observations before choosing either display.

For a small selection of variables, a pair plot draws their pairwise scatter plots and individual distributions. For example, use bill length in mm, flipper length in mm, body mass in g and species as a color group. About three to six numeric variables stay readable; adding many more creates a wall of small panels.

Investigate crowding and shared time trends

Log axes can spread positive values spanning large ratios, such as population and total energy use. Keep the scale visible in the labels. When one extreme point sets a huge axis range, inspect it and add a clearly labeled view of the bulk so the remaining pattern is visible.

Two variables can also trace a pattern because both grow over time. Color points by date or plot each series against time. That check can reveal whether the apparent relationship is mostly a shared trend.

A correlation coefficient summarizes an aspect of association. Choose the coefficient for the relationship you want to describe, and read it beside a plot and the number of usable observations.

Pearson measures a straight-line relationship

Pearson's coefficient is the centered cross-product total divided by the product of the centered lengths:

Its range is −1 to 1. The sign gives direction and the magnitude gives linear strength. Perfect positive or negative values mean all usable points lie on a straight line, with both variables having variation. The coefficient has no unit, is unchanged by swapping axes, and is unchanged by a positive conversion between measurement units.

A coefficient near zero means little linear association. For inputs −3, −2, −1, 0, 1, 2, 3 and outputs equal to their squares, Pearson correlation is exactly zero despite a perfect U-shaped relationship. Correlation also differs from slope: a shallow line can have a high correlation.

Rank methods describe ordering

Spearman correlation computes Pearson correlation on ranks and describes monotonic association: values tend to move consistently upward or downward, even along a curve. Kendall's tau compares concordant observation pairs, ordered the same way on both variables, with discordant pairs, ordered oppositely. Rank methods are less sensitive to extreme magnitudes than Pearson.

For the temperature–rental example, Pearson is 0.974, Spearman 0.976 and Kendall 0.929. The scatter plot still reveals the leveling above about 27 °C. A high coefficient can coexist with important curvature.

Compute a matrix carefully

cols = ["temp_c", "rentals", "distance_km"]
corr = df[cols].corr(method="spearman")
sns.heatmap(corr, annot=True, fmt=".2f",
            vmin=-1, vmax=1, center=0, cmap="vlag")

For a mixed DataFrame, df.corr(numeric_only=True) selects numeric columns; also exclude identifiers and category codes that lack a meaningful numeric relationship. Missing values are excluded pair by pair, so matrix cells may describe different subsets. Check each pair's usable count. The matrix is symmetric. Self-correlation is 1 where defined; a constant column has undefined correlation and a missing row, column and diagonal entry.

Check sensitivity to the data geometry

For inputs 1 through 10 and outputs 3, 1, 4, 2, 5, 3, 4, 2, 5, 3, Pearson is about 0.307. Adding the single point (30, 30) raises it to 0.938; Spearman becomes about 0.462. One point can dominate Pearson while the bulk retains a weak pattern.

Anscombe's quartet makes the same lesson visible: four datasets share a mean input of 9, mean output of 7.50, input variance of 11, correlation near 0.816 and fitted line $y=3.00+0.500x$, yet their plots show a cloud, a curve and different influential points. Report the value and count with any verbal strength label; fields have different conventions for labels such as "strong".

An association describes how variables occur together in the observed data. Its cause needs a separate explanation. Possible sources include a causal effect, reverse causation, a third variable, selection and chance.

Ask what could explain the link

A confounder influences both variables and creates or distorts their association. Hot weather increases ice-cream sales and swimming accidents. Reducing ice-cream sales would leave the weather-related risk in place. A mediator is different: it lies on the causal path between the variables.

Reverse causation reverses the proposed direction. Customers who visit help pages more often may have more failed orders because failures send them to those pages. The observed link gives no basis for expecting fewer failures after hiding help pages.

Separate subgroup patterns from the pooled result

Simpson's paradox occurs when the pooled comparison reverses the direction seen in each subgroup. The kidney-stone treatment data from a 1986 study illustrate the arithmetic:

A has the higher success share in both stone-size groups. B has the higher pooled share because A was used mostly in the harder large-stone cases. Pooled rates weight subgroup rates by their counts. Inspect composition and compare meaningful subgroups before interpreting a combined result.

Check time, selection and aggregation

Two unrelated series can both rise over the years and appear highly correlated. Plot them over time and inspect period-to-period changes or within-period relationships. The level relationship may mostly reflect their common time trend.

A selected range also changes what is visible. Among students selected for high entrance scores, the score range is narrow, and the score–grade correlation can be much weaker than among all applicants. Describe the selected population; its coefficient answers a different question from the coefficient for all applicants.

A correlation between city averages belongs to cities. Applying it to individual residents is the ecological fallacy. Aggregation can strengthen, weaken or reverse an association because the within-group pattern and between-group composition need not agree.

Treat a wide search as a source of hypotheses

With 20 numeric variables there are 190 distinct pairs. Searching those pairs and many segments gives chance patterns many opportunities to appear. Report how widely you explored, then identify a surprising pattern as a hypothesis for new data or a planned confirmatory analysis.

Use wording tied to the observations: "Orders placed in rain were associated with longer delivery times during this period." That statement records a relationship. A claim that rain caused the difference would also need evidence ruling out competing explanations, such as a different mix of routes or times.

A segment is a subset defined by a category, numeric band or period. A useful comparison shows how the whole distribution differs, with the amount of evidence behind each summary visible.

Put group sizes beside summaries

by_city = df.groupby("city")["minutes"].agg(
    ["count", "mean", "median", "std"]
)

The count here is the number of recorded durations. If some durations are missing, also report the number of orders and the missing share so the table's basis is clear. Named aggregations can give columns useful labels, such as orders=("order_id", "count") and median_min=("minutes", "median").

Consider these delivery times:

Tartu has the highest average but only two observations. One long delivery dominates its result. Small groups often appear at both ends of a ranking because their summaries swing more with individual observations. Set a minimum count before ranking and keep small groups visible as incomplete evidence.

Choose a display for the distribution

A box plot on a shared numeric axis compares medians, middle halves and flagged extremes. A violin plot adds estimated shape. With small groups, strip or swarm plots show each observation and make the sample size easier to see.

For histograms, use panels with the same axes or normalized overlays such as stat="density", common_norm=False. Count overlays mainly reveal which group contains more records. A density overlay lets each group's shape be compared on the same basis.

Two groups with the same mean may have very different tails. A city with mostly quick deliveries and a few very long ones can share a mean with a city of consistently moderate times. Compare spread, percentiles and a relevant threshold share alongside the center.

Name the baseline of a difference

The Tartu–Lyon mean difference in this sample is 8 minutes. Relative to Lyon, it is about 27.6%:

The baseline is Lyon's 29 minutes. Reversing the baseline changes the relative percentage, even though the absolute gap remains 8 minutes. Report the limited group counts with this descriptive calculation; the arithmetic alone cannot establish a stable city difference.

Check composition within groups

df.groupby(["city", "vehicle"])["minutes"].median().unstack()

This city-by-vehicle table can reveal whether a city difference comes from the mix of vehicles. A pivot table with the same grouping and median gives the same comparison. Inspect counts for these smaller cells too: splitting a group can make the evidence thinner even while clarifying the pattern.

Counts answer how many events occurred. Rates answer how often events occurred relative to their opportunities. When groups differ in size, the choice of denominator can change the apparent comparison.

Read a cross-tabulation in the intended direction

A cross-tabulation counts combinations of two categories:

pd.crosstab(df["region"], df["channel"], margins=True)

North has 30 app orders and 20 web orders; South has 60 app orders and 90 web orders. South has more app orders, but its larger order total changes the share comparison.

normalize="index" divides by row totals; normalize="columns" divides by column totals. normalize="all" uses the grand total. Read the question aloud before choosing the normalization. Row-normalized rows each sum to 1, while column-normalized columns do so.

Preserve unknown outcomes

A Boolean mean gives the share of True values. For a late-delivery flag, missing times need special treatment:

df["late"] = df["minutes"].astype("Float64") > 45
rates = df.groupby("city")["late"].mean()
known = df.groupby("city")["late"].count()

Nullable Float64 preserves missing times as missing flags. The grouped mean then estimates the late share among recorded times. Ordinary floating-point NaN > 45 is False, which would count unknown times as on time and lower the rate. State the known-outcome denominator and report missingness beside the rate.

Distinguish two kinds of percentage change

A late rate rising from 4% to 5% rises by 1 percentage point. The relative increase uses the original rate as baseline:

Write "rose by 1 percentage point, a 25% relative increase." A bare "rose by 1%" leaves the meaning unclear. For a rate based on a small cell, show its count too: 1 late order in 2 produces 50%, but a single additional on-time order changes it to one third.

Compare outcome shares with group size

If North supplies 25% of all orders but 40% of complaints, it is overrepresented among complaints. That compares its complaint share with its order share. It is a different question from the complaint rate within North, which requires North's complaint count divided by North's orders.

Use grouped bars, 100% stacked bars or a labeled heatmap to show proportions. A row-normalized region-by-channel table plotted as stacked bars compares channel mix within regions. Put the denominator in the title or axis label so readers can distinguish that mix from the regional composition of each channel.

Period totals depend on both activity and the time covered. Before describing a rise or fall, check period boundaries, missing dates and the normal seasonal pattern. Compare segments on a measure that fits their size.

Aggregate by calendar period

With parsed order timestamps, monthly counts can be built as follows:

monthly = (
    df.set_index("order_date")
      .resample("MS")["order_id"].count()
)

"MS" labels each calendar month by its first day. Use "ME" for month-end labels in current pandas. For a monthly city table, combine pd.Grouper(key="order_date", freq="MS") with the city grouping and use .size().unstack(). The method counts rows, so first establish that a row is an order.

Check exposure before comparing totals

Suppose there are 100 orders per day from August 1 through October 12. Monthly totals are 3,100 in August, 3,000 in September and 1,200 in the recorded part of October. The October total looks much lower, yet every observed day has the same activity.

Divide each total by its observed days: 31, 30 and 12. All three daily averages are 100. Compare such rates or omit incomplete periods. The denominator must represent actual coverage: an unrecorded day cannot safely be treated as a day with zero activity.

Even complete months have different lengths. They can also contain different numbers of busy weekdays. If Saturday activity is high, compare matching weekdays or rates that account for the weekday mix before interpreting a monthly difference.

Separate a cycle from a lasting change

Seasonality is a recurring pattern, such as a weekly or yearly cycle. daily.rolling(7).mean() smooths a seven-day window in a regularly spaced daily series. It makes a weekly cycle less prominent while retaining slower changes. Compare December with the previous December when the question is about year-over-year change in a seasonal activity.

A few unusually high days can look like a trend in a short window. Show enough history to understand the normal range of movement, and retain the unsmoothed values when they help explain an important event.

Check gaps and compare segment paths

A resampled count of zero can mean no recorded events or a collection outage. A sudden level change can reflect a new app version or a changed definition. Confirm coverage and collection history with the data source before treating those points as changes in behavior.

Plot each segment as a line on shared axes, with units and a consistent color per group. Large groups may dominate count plots. Shares, rates or an index relative to each group's starting value can make smaller groups' changes visible; label which measure is shown.

A cohort groups customers by when they started. Following March starters and April starters by time since their first order can separate changes within customers' histories from changes in the overall customer mix.

A useful finding connects a question to evidence. It gives the size of a pattern, the population and period it describes, and the limitations that affect its interpretation. Build the report around questions instead of the order in which charts were created.

Give a finding an evidence base

For example: "From January to September 2026, the median delivery time was 29 minutes; 8% of the 48,200 orders took longer than 45 minutes. The late share was highest on Friday and Saturday evenings: 14%, compared with 6% on other evenings."

This statement names the period, center, threshold, denominator and comparison. Use enough precision for the decision. "29 minutes" is easier to use than "29.4545 minutes" when the extra digits add no practical information. For a skewed duration, a median and an upper percentile often explain the experience better than a mean alone.

Separate what happened from why and what to do

A recommendation depends on assumptions about the cause and the action's effect. Keep those assumptions visible and name the follow-up evidence needed. An observation can be useful even while its explanation remains a hypothesis.

Make a chart explain one finding

Select the few charts that answer the questions and the surprises that matter. Keep other useful exploratory views in an appendix or notebook. Give a report chart a title that states its finding, such as "Late deliveries peak on Friday and Saturday evenings." Label axes with units, sort categories where appropriate and keep each segment's color consistent across charts. Include the data source and period.

Report contradictory evidence too. If a pooled chart suggests improvement but one large segment worsened, the segment result matters to the interpretation. Selecting only charts that support an expected story would hide information the reader needs.

Organize a compact report

A clear order is question and context; data source, period, row count and row meaning; quality issues and cleaning decisions; three to five key findings with evidence; limitations; next steps. Each finding should answer a question or explain an important surprise.

Limitations identify what the data cover, what was excluded and why, how much is missing, and the association-based nature of the patterns. For example, disclose 37 canceled orders removed from a completed-delivery comparison. Next steps can include checking a hypothesis on new data, collecting a missing variable, fixing a source-system issue or investigating a question for later modeling.

A shared analysis needs both readable results and a route back to the code and inputs that produced them. Prepare a final notebook whose order, outputs and supporting files agree.

Clean and execute the final notebook

Remove dead and duplicate cells, keep dependencies in run order and add Markdown that explains decisions between code sections. Preserve the more experimental work separately when it is useful. Restart the kernel and run all cells before every export. Inspect both errors and changed outputs: a successful run can still reveal that an old written number is stale.

Generate reported numbers from code:

s = df["minutes"]
print(f"Median delivery time: {s.median():.0f} minutes")

A number typed into a paragraph can remain unchanged after the data refresh. Keep the displayed statement linked to the calculation so the next run updates it.

Export reports, figures and tables

An HTML export can be read in a browser without a Python environment:

jupyter nbconvert --to html --execute report.ipynb

The execute option runs the notebook before conversion. Check the resulting report as well as the notebook. Publishing tools such as Quarto offer another route to a readable report.

Save figures and tables from code so reruns refresh the files:

fig.savefig("figures/late_by_hour.png", dpi=200,
            bbox_inches="tight")
summary.to_csv("output/city_summary.csv")

Create the destination folders as part of project setup. A screenshot preserves one moment's output but leaves no automatic link to the next calculation.

Include the provenance and run instructions

Record the data source, extraction date, input file or version, row count, filters, software versions and random seeds. These details explain which input and settings produced a finding. Collect changing settings near the top of the notebook so a monthly update has one clear entry point.

A project can separate raw and processed data, notebooks, figures and exported tables. Include a README that states the purpose, expected working directory and run steps, plus a package environment file. Keep notebook, instructions and environment together when sharing the rerunnable analysis. If inputs need controlled access, state how an authorized reader obtains the matching data version.

Check what the outputs reveal

Before sharing, inspect every output cell and export for personal information. Preview tables can expose names, email addresses and exact locations even when the final chart is aggregated. Remove, mask or aggregate identifying columns as appropriate, and follow the agreements and data-protection rules that apply to the dataset.