Getting Started with pandas

A Series keeps a sequence of values together with labels. Those labels give the values meaning: a rain reading belongs to a day, and a population belongs to a city. Import pandas and NumPy using their usual short names.

import pandas as pd
import numpy as np
rain = pd.Series([4.5, 3.0, 6.2],
                 index=["Mon", "Tue", "Wed"], name="rain_mm")

Read values and labels

The values are 4.5, 3.0 and 6.2; the index is Mon, Tue and Wed. Without index=, pandas supplies labels 0, 1 and 2. A dictionary supplies both at once: pd.Series({"Lima": 10, "Oslo": 5, "Pune": 7}) takes its labels from the keys, in dictionary order.

Attributes describe the Series without parentheses: rain.index holds the labels, rain.dtype is float64, rain.name is "rain_mm" and rain.size is 3. Use rain.to_numpy() when a NumPy array is needed. The older .values returns different kinds of array for different dtypes; text data can give a pandas array.

A dtype tells pandas what kind of values it stores. Common types are int64 for whole numbers, float64 for decimals, bool for True/False, str for text and datetime64 for dates and times. pandas 3 infers text as str; older examples often display object.

Work on the whole Series

For s = pd.Series([10, 20, 30], index=["a", "b", "c"]), s * 2 produces values 20, 40 and 60. s > 15 produces False, True and True. Both keep the labels. These operations are vectorized: one expression works on the whole Series without a Python loop.

Look up a label with s["b"], which gives 20. Look up a position with s.iloc[0], which gives 10. The expression "b" in s checks the index; 20 in s is False. Even an integer in square brackets is a label: with index [2, 0, 1], s[0] means label 0, not the first entry. With text labels, s[0] raises KeyError in pandas 3.

Match labels in arithmetic

a = pd.Series([1, 2, 3], index=["x", "y", "z"])
b = pd.Series([10, 20, 30], index=["y", "z", "w"])
a + b

The shared label y combines 2 and 10 to give 12.0; z combines 3 and 20 to give 23.0. Labels w and x appear in only one input, so they receive NaN. The result includes all four labels. Adding by visual row position would incorrectly give 11, 22 and 33.

Use a.add(b, fill_value=0) when an absent value should count as zero. Its results are w 30.0, x 1.0, y 12.0 and z 23.0. Related methods include sub, mul and div.

Recognize missing values

A missing float or str value uses NaN; a missing datetime uses NaT; a missing nullable integer uses <NA>. pd.Series([1, None, 3]) becomes float64, displaying 1.0, NaN and 3.0, because ordinary int64 cannot hold NaN.

A DataFrame is a two-dimensional table. Each row records one observation, such as an order; each column holds one variable, such as a quantity. Columns can have different dtypes, but they share the same row index.

Build a table from Python data

df = pd.DataFrame({
    "product": ["Notebook", "Desk lamp", "Pen set", "Mug"],
    "quantity": [3, 1, 2, 4],
    "unit_price": [2.50, 18.00, 4.75, 6.00]
})

Each dictionary key becomes a column name; each list supplies that column's values. The lists must have equal lengths so that each row has a place in every column. Here the index labels are 0 through 3. Lists of different lengths raise a ValueError.

A list of dictionaries instead supplies one dictionary per row. pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "c": 4}]) creates columns a, b and c. The missing b in the second row and missing c in the first become NaN, so those columns use float64. A two-dimensional NumPy array can also supply values, with columns=[...] naming the columns.

Read the table's dimensions

df.shape is (4, 3): four rows and three columns, in that order. len(df) is 4, while df.size is 12 because it counts cells. df.index holds row labels, df.columns holds column labels and df.dtypes gives one dtype per column. These are attributes: df.shape() tries to call a tuple and raises TypeError. df.T swaps rows and columns.

Keep a Series or keep a table

The inner list requests a collection of columns. Without it, df["product", "unit_price"] supplies one tuple key, which does not match either ordinary column name and raises KeyError.

Square brackets also work when names contain spaces or clash with methods. Dot notation such as df.quantity only works for suitable names. A column called size needs df["size"], because df.size means the table's cell count. Create columns with square brackets too: df["total"] = 5 adds a column, while df.total = 5 does not.

Choose an axis

For numbers = pd.DataFrame({"a": [1, 2], "b": [3, 4]}), numbers.sum() adds down the rows, giving a 3 and b 7. This is axis=0, also called "index". numbers.sum(axis=1) adds across each row, giving 4 and 6; this axis is also called "columns". Name the result you need first: one total per column or one total per row.

Taking one row creates a Series whose labels are the column names. That Series needs one dtype able to hold every cell in the row. A row mixing integers and floats becomes float64; a row mixing text and numbers becomes object. The original columns still retain their separate types.

Row labels can carry meaning, such as an order number, or simply label the current rows. Choosing a useful index makes lookups and later results easier to interpret. Changing labels returns a new table, so keep the result with an assignment.

Move an ID into the index

orders = pd.DataFrame({
    "order_id": [1001, 1002, 1003],
    "product": ["Notebook", "Desk lamp", "Pen set"],
    "unit_price": [2.50, 18.00, 4.75]
})
orders_by_id = orders.set_index("order_id")

The original table has a default RangeIndex: 0, 1 and 2. The new table has labels 1001, 1002 and 1003. order_id has moved out of the columns into the index. orders_by_id.loc[1002] now identifies the Desk lamp row by its ID.

Calling orders.set_index("order_id") without saving its result leaves orders unchanged. Reassignment, such as orders = orders.set_index("order_id"), keeps the new structure. In pandas 3.1, inplace= is deprecated for these label-changing methods and for sorting; use assignment.

Move labels back into a column

orders_by_id.reset_index() restores order_id as an ordinary column and supplies a new RangeIndex. An unnamed index becomes a column called index. This preserves the old row identity.

reset_index(drop=True) discards the old labels. For an index [10, 20, 30], plain reset_index() keeps those numbers in an index column; reset_index(drop=True) loses them and uses 0, 1 and 2. After filtering or sorting, choose whether the old labels are useful before discarding them. Renumbering the index does not change the data values or undo a sort.

Rename or remove labels

renamed = orders.rename(columns={"unit_price": "price"})
short = orders.drop(columns=["product"])
remaining = orders.drop(index=[0, 1])

rename maps old labels to new ones and keeps names not listed. By default a misspelled old name is ignored, so inspect .columns if a rename seems to have done nothing. To replace every column name, assign a full list to orders.columns; its length must match the column count.

drop(columns=...) removes columns; drop(index=...) removes rows by label. A label that is absent raises KeyError. After sorting, label 0 may no longer be the first row: dropping it still removes the row labeled 0.

Select a list to choose column order: orders = orders[["order_id", "unit_price", "product"]]. Columns absent from that list are removed from the result.

Check that an ID really identifies one row

Index labels can repeat. Check orders_by_id.index.is_unique before treating them as unique IDs. If label 1002 occurs twice, .loc[1002] returns both rows as a DataFrame. With unique labels, the same single-label lookup returns a Series.

A meaningful index also improves summaries. In a weather table indexed by day, weather["temp_c"].idxmax() reports a day such as "Thu"; with a default index it reports a row number.

A CSV stores a table as text: records occupy lines and a delimiter separates fields. The first line often contains column names. Loading the file correctly means identifying both its layout and the meaning of its values.

Load the orders table

orders = pd.read_csv("orders.csv")

A relative path is resolved from Python's current working directory. A wrong location raises FileNotFoundError. An absolute path or a URL can also be passed to read_csv.

The examples use this file. The final row has an empty quantity field.

order_id,date,product,category,quantity,unit_price
1001,2026-03-02,Notebook,Stationery,3,2.50
1002,2026-03-02,Desk lamp,Home,1,18.00
1003,2026-03-03,Pen set,Stationery,2,4.75
1004,2026-03-04,Mug,Home,4,6.00
1005,2026-03-04,Backpack,Bags,1,35.00
1006,2026-03-05,Notebook,Stationery,,2.50

With defaults, the result has shape (6, 6) and index labels 0 through 5. pandas infers types: order_id is int64, unit_price is float64, and quantity also becomes float64 because its blank becomes NaN. The other columns, including date, are str. A date-looking string is still text until parsed.

Match the file's layout

For a file whose lines are a;b, 1,5;2 and 3,0;4, use pd.read_csv("file.csv", sep=";", decimal=","). Column a then contains 1.5 and 3.0. Using the default comma separator produces one column named a;b, a sign to check the delimiter.

index_col="order_id" makes that column the row index during loading. thousands="," reads a field such as "1,200" as 1200; in a comma-separated file the field needs CSV quoting so that its internal comma is not another delimiter.

Preserve meaning while inferring types

Codes such as "001" are identifiers, not amounts. dtype={"id": str} preserves their leading zeros. If pandas first converts them to 1, converting that number back to text cannot recover how many zeros the file contained.

Empty fields and common markers such as NA, N/A, NULL, NaN and None are missing by default. Add a file's special marker with na_values=["-"]. These are loading rules, so check the file's conventions when a code looks like a missing marker.

Use parse_dates=["date"] to read the example dates as datetimes. In pandas 3 these dates use microsecond resolution, datetime64[us]. UTF-8 is the default text encoding. A UnicodeDecodeError calls for checking the file's encoding and supplying the matching option, such as encoding="latin-1".

A successful read means pandas could parse the file. It does not establish that the columns, types and missing values match what the table is meant to represent. Inspect the new DataFrame before calculating from it.

Check visible rows and dimensions

orders.head(3)
orders.tail()
orders.shape
orders.columns.tolist()

head() shows five rows by default; head(3) shows three. tail() checks the end, where an export may contain different records. sample(3, random_state=0) gives a repeatable random sample of three rows. In a notebook, a cell's final expression displays automatically; use print elsewhere.

The orders file should contain six rows and six columns. A preview should show named columns and the first order as data. If the first record appears among the column names, the file probably lacks a header and needs header=None. If everything occupies one wide column, check sep.

Column names match exactly, including case and spaces. orders["Product"] fails when the name is product. Inspecting the list makes trailing spaces easier to spot. orders.columns = orders.columns.str.strip() removes spaces at the ends of header names.

Read the structure with info

orders.info() prints the index range, each column's name, count of present values, dtype and memory use. For the six-row example, quantity has five non-null values; the other columns each have six. The dtypes are two float columns, one integer column and three str columns. orders.dtypes gives a compact type list.

A numeric-looking column with dtype str deserves inspection: a token such as "unknown", a unit such as "12 kg" or an unhandled thousands separator can block numeric inference. Dates also remain text unless parsed. Decide the needed type from the column's meaning, not only its appearance in a preview.

Count gaps and repeated values

orders.isna().sum()
orders.isna().any(axis=1).sum()
orders.nunique()

isna() makes a boolean table: True marks a missing cell. Summing down each column counts those cells. Quantity has one missing value and every other column has zero. To count affected rows, .any(axis=1) first checks whether each row has any gap, then .sum() counts those rows. Adding all column counts instead could count one row several times.

nunique() counts distinct present values: the orders have six IDs, four dates, five products, three categories, four quantities and five prices. Compare the ID count with the row count to check uniqueness, along with its missing count. orders.duplicated(subset="product") marks the second Notebook row, label 5, as repeated. A repeated product is legitimate here; a repeated order ID would need investigation.

Read a quick numeric overview

orders.describe() reports count, mean, standard deviation, minimum, quartiles and maximum for numeric columns. Quantity's count is five, so its mean summarizes five recorded quantities. Check plausible ranges and units as well as types: a column can be numeric and still contain an input error. Keep the observed row count, column count, types, missing counts and ID check together when reviewing a new load.

A column's dtype determines which operations make sense. A text column containing "12" still stores text; it cannot be averaged as a numeric column. Convert deliberately and assign the result back to the table.

Choose strict conversion or coercion

astype converts a Series to a named type, such as int, float, str or "Int64". It fails when a value cannot be represented in the requested type.

raw = pd.Series(["12", "7", "n/a", "3.5"])
numeric = pd.to_numeric(raw, errors="coerce")

raw.astype(float) raises ValueError at "n/a". With errors="coerce", the output values are 12.0, 7.0, NaN and 3.5. The failed token becomes missing instead of stopping the conversion. Compare raw.isna().sum() with numeric.isna().sum(): zero missing values became one, identifying a value that needs checking.

Coercion is a conversion policy, not evidence that every new gap is harmless. Use the before-and-after counts to find losses. Store the conversion in a new variable or column when the raw text is useful for tracing a failed value.

Keep missing whole numbers

The orders quantity column contains whole numbers plus one NaN, so it is float64. Plain astype(int) fails because NaN has no ordinary integer representation.

orders["quantity"] = orders["quantity"].astype("Int64")

The capital I matters: nullable Int64 keeps the gap as <NA> while storing the known values as whole numbers. This choice fits counts with missing entries. It does not make a genuinely fractional quantity into a valid whole-number count.

Text thousands separators also need handling. pd.Series(["1,200", "950"]).str.replace(",", "").astype(int) yields 1200 and 950. When the file itself uses that convention, thousands="," can handle it during reading.

Convert date text

orders["date"] = pd.to_datetime(orders["date"])
orders["date"].dt.day_name()

The conversion enables .dt.year, .dt.month, .dt.day and .dt.day_name(). For the six order dates, the day names are Monday, Monday, Tuesday, Wednesday, Wednesday and Thursday. Running pd.to_datetime(orders["date"]) alone produces a converted Series but leaves the original column as text. The assignment is what keeps it.

Store repeated categories

orders["category"].astype("category") stores repeated labels through a set of allowed categories. On large tables this usually saves memory. Assign the result to keep it, just as for other conversions.

Category dtype changes some operations. Its categories are unordered by default, so min() and max() raise TypeError. A category column's value_counts() also lists known categories absent from a filtered selection with count zero. Choosing category dtype is useful for repeated labels, but it is a change in behavior as well as storage.

Labels let you ask for the row or column by its meaning. With loc, an order number identifies a row even if the table has been sorted. The selection has a row part followed by a column part: df.loc[rows, columns].

Start from a meaningful index

The orders file has six rows with IDs 1001 through 1006. Create orders_by_id = orders.set_index("order_id"). Its columns, in order, are date, product, category, quantity and unit_price. The row labels are order numbers, not positions 0 through 5.

orders_by_id.loc[1003]
orders_by_id.loc[1004, "product"]

The first expression returns order 1003 as a Series: date "2026-03-03", product "Pen set", category "Stationery", quantity 2.0 and price 4.75. The Series is named 1003. The second returns one cell, "Mug".

A missing label raises KeyError. orders_by_id.loc[0] fails: having a first row at position 0 does not create a label 0.

Select collections of labels

orders_by_id.loc[[1001, 1005], ["product", "unit_price"]]

Lists select labels in the order requested. This gives two rows and two columns: Notebook at 2.5, then Backpack at 35.0. A list preserves a table-shaped result even when it contains one label: .loc[[1003]] is a one-row DataFrame.

A colon alone means all labels along that axis. Leaving out the column selector also requests all columns.

Include both ends of a label slice

orders_by_id.loc[1002:1004, "product":"quantity"]

On this ordered index, the rows are 1002, 1003 and 1004. The column slice includes product, category and quantity. Both the row stop label and the column stop label are included, so the shape is (3, 3). Applying Python's usual exclusive-stop rule here would omit an intended row and column.

Labels survive sorting

A default index can hide the distinction between label and position. Suppose sorting scores gives row labels 3, 0, 2, 1, 4. .loc[0] still selects label 0, now the second displayed row. .iloc[0] selects the top displayed row, label 3. Sorting moves a row with its label; it does not silently renumber the index.

Combine a condition with column selection

The row selector can be a boolean Series:

orders.loc[orders["unit_price"] > 5, ["product", "unit_price"]]

This keeps Desk lamp at 18.0, Mug at 6.0 and Backpack at 35.0, and returns only the two requested columns. The mask decides which rows qualify; the column list decides what to show. The same two-axis form can target cells for an update by putting the selection on the left side of an assignment.

Use iloc when a request concerns where a row or column sits: the first three rows, the last column or a particular displayed cell. Its selectors are integer positions, independent of the index labels.

Count from zero

orders.iloc[0] selects the first row and orders.iloc[-1] the last. A negative position counts backward from the end. Two positions select a cell: orders.iloc[2, 2] is "Pen set" in the original orders table, where product is column position 2.

The indexed table orders_by_id has columns date, product, category, quantity and unit_price. Moving order_id into the index changed its column positions, so product is now position 1. Position-based code depends on this layout.

Stop before the slice endpoint

orders_by_id.iloc[1:3, 0:2]

The selected row positions are 1 and 2, containing orders 1002 and 1003. The column positions are 0 and 1, date and product. Position 3 and column 2 are excluded, giving shape (2, 2). An empty start means the beginning; an empty stop means the end.

Lists request particular positions, including a mixture of positive and negative ones. orders.iloc[[0, -1], [2, 5]] selects product and unit_price for the first and last orders. Both happen to be Notebook at 2.5.

Predict the result's shape

A scalar selects one item along an axis and removes that dimension. A slice or list keeps that axis, even if it selects just one item. This matters when the next operation expects a table instead of a Series.

Handle the end of a table

The six-row orders table has valid nonnegative row positions 0 through 5. orders.iloc[10] raises IndexError. A slice can extend beyond the table: orders.iloc[4:100] simply returns the last two rows. A single missing position is an invalid item request, while a slice describes a range that is clipped at the available rows.

Choose labels or positions deliberately

For a single cell, at uses labels and iat uses positions. orders.at[2, "product"] and orders.iat[2, 2] both give "Pen set" before the rows are reordered.

Choose loc when you know an ID or a column's meaning. Choose iloc when the location itself matters. To combine first rows with a column name, orders.columns.get_loc("quantity") returns its position, 4, allowing orders.iloc[0:3, 4].

Bare square brackets can switch roles: orders["product"] selects a column, but orders[2:4] slices row positions 2 and 3. Explicit loc and iloc make that choice easier to read.

Filtering selects observations that satisfy a question, such as which orders belong to Home and cost more than 10 per item. First create a boolean mask, a Series with one True or False per row. Then use it to select the matching rows.

Build and apply a mask

is_stationery = orders["category"] == "Stationery"
stationery = orders[is_stationery]

The mask is True, False, True, False, False and True. The selection keeps labels 0, 2 and 5, corresponding to orders 1001, 1003 and 1006. Filtering preserves those labels; it does not renumber them. reset_index(drop=True) supplies consecutive labels if needed.

Comparison operators include ==, !=, <, <=, > and >=. Text comparisons match exactly: "home" differs from "Home", and spaces can also prevent a match.

Combine conditions row by row

is_home = orders["category"] == "Home"
is_expensive = orders["unit_price"] > 10
orders[is_home & is_expensive]

Only the Desk lamp qualifies: its category is Home and its price is 18.0. The Mug is Home but costs 6.0. Backpack costs 35.0 but belongs to Bags.

When writing comparisons directly, put each in parentheses:

orders[(orders["category"] == "Home") &
       (orders["unit_price"] > 10)]

Use & for and, | for or and ~ for not. Parentheses keep each comparison together before pandas combines the masks. Without them, Python's precedence rules can try to combine a string and a Series with & and raise TypeError.

Python's and, or and not expect one truth value, not a column of truth values. Using them on masks raises the ambiguous-truth-value ValueError. A plain if orders["quantity"] > 1: has the same problem. Use .any() to ask whether any row matches, or .all() to ask whether every row matches.

Count matches and proportions

is_stationery.sum() is 3 because True counts as 1 and False as 0. is_stationery.mean() is 0.5, the fraction of six orders that match. Multiply by 100 to express that as 50%.

Treat missing values explicitly

With the original float64 quantity column, comparisons involving NaN give False except !=, which gives True. quantity >= 1 keeps five rows; quantity < 1 keeps none. The blank quantity is in neither selection. Yet quantity != 3 includes that blank row, so unequal-to does not mean known-and-different.

After conversion to nullable Int64, comparison with <NA> produces <NA>, and filtering leaves those rows out. Use isna() or notna() when the question concerns gaps.

For selected columns as well as rows, use orders.loc[is_home, "product"]. It returns Desk lamp and Mug. A named mask can be reused for inspecting, counting and updating the same set of rows.

Some filters have familiar shapes: membership in a list, a numeric interval, a missing value or a text fragment. Helper methods express these tests directly and still produce masks that can be combined with other conditions.

Test membership and ranges

orders[orders["category"].isin(["Home", "Bags"])]
orders[orders["unit_price"].between(4.75, 18)]

isin matches any listed value. The first selection contains orders 1002, 1004 and 1005. ~orders["category"].isin(["Home"]) reverses membership and keeps Stationery and Bags rows. A Python in expression does not make this row-by-row membership mask.

between includes both endpoints by default. The second selection includes Desk lamp at 18.0, Pen set at 4.75 and Mug at 6.0. Use inclusive="neither" to exclude both endpoints, "left" to include only the low endpoint or "right" to include only the high endpoint. Choose the endpoint rule from the question being asked.

Find missing and present entries

orders[orders["quantity"].isna()] keeps order 1006. notna() does the reverse, keeping recorded quantities. Do not test equality with np.nan: NaN is unequal even to itself, so quantity == np.nan does not find the missing row.

Search inside text

orders[orders["product"].str.contains("note", case=False, na=False)]

str.contains tests whether each text value contains a match. Ignoring case finds both Notebook rows. na=False makes missing text count as no match. The search treats its pattern as a regular expression by default; for a literal text fragment with punctuation, regex=False avoids interpreting that punctuation as a pattern. str.startswith and str.endswith test the beginning and end.

Write a filter as a query

orders.query("quantity >= 2 and unit_price < 5")

This keeps orders 1001 and 1003. Within a query string, column names stand for columns and and, or and not are allowed. The string is a pandas query expression; it is not the same syntax as combining Series directly in ordinary Python.

A Python variable needs @:

limit = 5
orders.query("unit_price > @limit")

The result contains Desk lamp, Mug and Backpack. A column name containing a space needs backticks inside the query, as in orders.query("`unit price` > 5").

The same question can be written as a mask, a helper call or a query. Prefer the form that exposes the rule clearly. For an interval, between puts the bounds and inclusion rule together. For a set of allowed labels, isin avoids repeating equality tests. When a filter behaves unexpectedly, inspect the mask and exact column values before changing the condition.

An update needs to target the table you intend to change. In pandas 3, a selection behaves as a separate object under Copy-on-Write, so changing a selected object does not change its parent table.

Put the whole target in one loc assignment

sale = orders.copy()
is_home = sale["category"] == "Home"
sale.loc[is_home, "unit_price"] *= 0.9

The mask selects two Home rows. The column selector targets their prices. Multiplication by 0.9 keeps 90% of each price, reducing 18.00 to 16.20 and 6.00 to 5.40. Other rows are untouched. The expanded form reads the same selection on the right side and assigns its multiplied values on the left.

Here sale is a separate table used for a price change, leaving orders available for comparison. To update the original table, put orders.loc[...] itself on the left of the assignment.

Avoid chained assignment

orders[orders["category"] == "Home"]["unit_price"] = 0

This uses two indexing steps. The first produces a selected object; the second assigns to that object. In pandas 3 it never updates orders, and a ChainedAssignmentError warning points out the problem. The same issue affects orders["unit_price"][1] = 0.

Use orders.loc[orders["category"] == "Home", "unit_price"] = 0 when the intended change is to the original table. The table, matching rows and column are all specified in one assignment.

Understand selected tables

After home = orders[orders["category"] == "Home"], changing home["unit_price"] changes only home. Later changes to orders do not change home either. The objects may share storage internally until a write is needed, but their visible values behave independently.

Older pandas examples may show SettingWithCopyWarning and use .copy() after selection to avoid uncertain updates. pandas 3 removed that warning and makes selection independence consistent. An explicit .copy() is still useful for stating intent, but a selection already behaves independently for updates.

Match the right-side values to the target

A scalar fills all selected cells. A list needs exactly one value per selected row and fills them in selection order. A Series aligns by index label:

new_prices = pd.Series([16.2, 5.4], index=[1, 3])
sale.loc[is_home, "unit_price"] = new_prices

The Home rows have labels 1 and 3, so they receive 16.2 and 5.4. If a selected label were missing from the Series, its assigned value would be NaN; rows outside the selection would remain unchanged. Equal lengths alone do not establish a Series-to-row match.

Inspect sale.loc[is_home, "unit_price"] after the change and compare it with orders.loc[is_home, "unit_price"]. The mask's .sum() is 2, which gives the expected number of targeted rows for this discount.

New columns turn recorded data into quantities you can analyze. A calculation between columns works row by row, while the index keeps each result attached to its observation.

Calculate revenue

orders["revenue"] = orders["quantity"] * orders["unit_price"]

The six revenues are 7.5, 18.0, 9.5, 24.0, 35.0 and NaN. For the first row, three items at 2.50 each give 7.50. The last row has an unknown quantity, so its revenue is unknown too. Assigning to a new name adds a column at the right; assigning to an existing name overwrites that column.

Column arithmetic is vectorized. It expresses one rule for the whole column and avoids a Python loop over rows. On large tables this is much faster than for or iterrows, and the expression makes the relationship between inputs clear.

Use the recorded values in a total

orders["revenue"].sum() skips NaN and gives 94.0, the total of the five known revenues. .mean() gives 18.8 because the denominator is five recorded revenues. Dividing 94.0 by all six rows would silently treat the missing revenue as if it contributed a known zero.

To calculate percentage shares:

orders["share_pct"] = (
    orders["revenue"] / orders["revenue"].sum() * 100
).round(1)

The shares are 8.0, 19.1, 10.1, 25.5, 37.2 and NaN. These are shares of the known total. The missing entry remains missing through arithmetic.

Return a table with assign

assign returns a new DataFrame with added or replaced columns, leaving its input unchanged. It works naturally inside a chain:

with_tax = orders.assign(
    tax=lambda d: (d["revenue"] * 0.2).round(2)
)

The lambda reads the table at that point in the chain. It gives tax values 1.5, 3.6, 1.9, 4.8, 7.0 and NaN. In pandas 3, pd.col("revenue") * 0.2 is another way to refer to a column in assign.

Decimal calculations can expose tiny binary floating-point errors: 9.5 times 0.2 can be stored as 1.9000000000000001. .round(2) gives a useful presentation value. Avoid exact equality tests on calculated decimals when small representation errors are possible.

Know how assignment matches rows

orders["home_price"] = orders.loc[
    orders["category"] == "Home", "unit_price"
]

The assigned Series has labels 1 and 3. Those rows receive 18.0 and 6.0; every other row receives NaN. Series assignment matches labels, even when its length differs from the table's. A plain list has no labels and must contain exactly one value per row. A scalar, such as orders["currency"] = "EUR", fills every row.

Derive values from text or dates

The .str and .dt accessors provide vectorized column operations too. orders["product"].str.upper() yields uppercase names, and .str.len() gives lengths 8, 9, 7, 3, 8 and 8. Assign such results to a new column when both the original and derived values are useful.

Transforming a column can mean applying a numeric rule, assigning a label or translating existing labels. Choose a method whose behavior matches the rule, especially for values the rule does not name.

Turn a condition into labels

orders["price_band"] = np.where(
    orders["unit_price"] >= 5, "high", "low"
)

np.where chooses its second argument where the condition is True and its third where it is False. For prices 2.50, 18.00, 4.75, 6.00, 35.00 and 2.50, the labels are low, high, low, high, high and low. The >= includes a price exactly equal to 5.

Translate all values or change a few

codes = {"Stationery": "ST", "Home": "HM", "Bags": "BG"}
orders["category_code"] = orders["category"].map(codes)

Dictionary map gives ST, HM, ST, HM, BG and ST. A value absent from the dictionary becomes NaN. With only {"Stationery": "ST"}, both Home and Bags become missing. Use a complete mapping when every category needs a translated code.

replace changes listed values and leaves unlisted ones unchanged. orders["category"].replace({"Bags": "Luggage"}) keeps Stationery and Home while renaming Bags.

Apply a custom function

A Series can pass each value to a function through map or apply. orders["product"].apply(len) returns each product name's length. Pass len itself; len() would call it immediately without a value.

A lambda can express a small value rule: orders["unit_price"].apply(lambda p: "budget" if p < 5 else "standard"). This returns budget, standard, budget, standard, standard and budget.

For logic needing several cells in one row, DataFrame apply accepts axis=1:

orders.apply(
    lambda r: r["quantity"] * r["unit_price"], axis=1
)

The function reads each row's quantity and price. For this arithmetic, direct column multiplication is clearer and much faster on a large table. Use vectorized expressions or np.where when they can express the calculation; reserve row-wise functions for logic that needs them.

Use built-in numeric transformations

round(n) rounds to n decimal places, abs() gives absolute values and clip(lower=..., upper=...) caps values at limits. orders["unit_price"].clip(upper=20) changes 35.0 to 20.0 and leaves the other five prices unchanged. It does not remove the expensive row.

These transformations return new values. Assign them to a new column to preserve the input for checking, or overwrite the original when that is intended. Inspect a few transformed rows or run value_counts() to check that new labels and their counts fit the rule.

Sorting changes the order in which observations appear. It keeps each row's values together and carries the row label along. Decide what takes priority and how tied values should be ordered before taking a top or bottom selection.

Sort one column

scores = pd.DataFrame({
    "name": ["Ana", "Ben", "Chen", "Dara", "Eli"],
    "points": [88, 92, 88, 75, 92]
})
ordered = scores.sort_values("points")

Smallest comes first by default. Dara's 75 is first, followed by the two 88s and then the two 92s. Row labels move with the records. Among ties, the default sorting algorithm does not guarantee the original order; the displayed labels can read 3, 0, 2, 1, 4, but tied rows must not be assumed to keep that order.

Use ascending=False for largest first. sort_values returns a table, leaving scores unchanged unless its result is assigned.

Choose how ties are resolved

scores.sort_values(
    ["points", "name"], ascending=[False, True]
)

The first key has priority. Points decrease; names increase within equal points. The order is Ben, Eli, Ana, Chen and Dara. Sorting names first would answer a different question because it would make names the primary key.

When one-column ties should preserve input order, use scores.sort_values("points", kind="stable"). Stability keeps tied rows in their existing sequence. A secondary key instead gives ties their own explicit ordering rule.

Place missing values and text

Missing values go last by default in both ascending and descending sorts. pd.Series([3.0, np.nan, 1.0]).sort_values() gives 1.0, 3.0, then NaN. na_position="first" moves the missing entry to the beginning; reversing sort direction alone does not do that.

Text sorting follows character codes. "Zebra" sorts before "apple" because uppercase and lowercase letters occupy different ranges. Consistently written names such as Bags, Home and Stationery sort in the expected alphabetical order.

Sort labels or renumber them

sort_index() orders the row labels, not the data values. After sorting by points, it can restore the original default-label order. sort_index(ascending=False) reverses label order. By contrast, reset_index(drop=True) leaves the current row order in place and replaces its labels with 0, 1, 2 and so on.

Select the largest or smallest rows

scores.nlargest(2, "points") returns Ben and Eli, both with 92. nsmallest takes low values instead. If the cut falls inside a tie, decide whether to keep every tied row. scores.nlargest(1, "points", keep="all") returns both 92s, even though n is 1. The number requested is then a cutoff, not a promise about the final row count.

A rank records where a value belongs in sorted order without moving its row. It is useful when each observation needs a place number alongside its original data. Ties need a rule because several values can occupy the same part of that order.

Choose the direction

scores["points"].rank() ranks from smallest to largest. Use ascending=False for a results table where high scores should get the best ranks. The tie method then decides which place number each tied group receives.

The example scores in row order are Ana 88, Ben 92, Chen 88, Dara 75 and Eli 92. In descending order, the 92s occupy places 1 and 2, the 88s places 3 and 4, and 75 place 5. The default method="average" assigns each tied value the mean of its occupied places. The returned values, in original row order, are 3.5, 1.5, 3.5, 5.0 and 1.5.

Pick a tie method

With min, two winners at rank 1 mean the next score starts at rank 3. With dense, the next distinct score starts at rank 2. first breaks a tie even when the scores themselves provide no basis for preferring one person; it follows input order.

Rank results are floats, allowing values such as 1.5. When a chosen tie method gives whole-number ranks and none are missing, .astype(int) converts them for display. An ordinary integer dtype cannot hold a missing rank.

Express ranks as fractions

For the default average method and non-missing values, rank(pct=True) divides each rank by the number of ranked values. Ranking the five example scores ascending gives 2.5, 4.5, 2.5, 1.0 and 4.5. Dividing by five gives 0.5, 0.9, 0.5, 0.2 and 0.9.

The tied highest values receive 0.9, not 1, because their average rank is 4.5. A percentile rank describes rank relative to the ranked values; it is not a percentage of the score itself.

Handle missing ranks and store results

Missing values retain NaN ranks by default. With revenues 7.5, 18.0, 9.5, 24.0, 35.0 and NaN, descending ranks are 5, 3, 4, 2, 1 and NaN. na_option="bottom" puts the missing entry after recorded values, giving it rank 6.

scores["place"] = scores["points"].rank(
    method="min", ascending=False
)

The rows remain Ana, Ben, Chen, Dara and Eli. The new place column records 3, 1, 3, 5 and 1. Sort separately if the display should show winners first.

A descriptive statistic compresses many observations into one useful quantity. Choose a statistic that fits the question, and read its non-missing count so you know how much data supports it.

Compute centers and spread

weather = pd.DataFrame({
    "temp_c": [14.0, 17.5, 16.0, 21.0, 19.5, 12.5, 15.0],
    "rain_mm": [0.0, 2.4, 0.0, 0.0, 5.1, 12.3, 0.8]
}, index=["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"])

For weather["temp_c"], sum() is 115.5, mean() is 16.5 and median() is 16.0. The minimum is 12.5 and maximum 21.0, giving a range of 8.5 °C. idxmax() returns the label "Thu", while idxmin() returns "Sat": they identify observations instead of returning the extreme values themselves.

std() and var() measure spread. pandas uses the sample denominator by default, ddof=1, dividing squared deviations by one less than the count. Here standard deviation is about 3.028 °C and variance about 9.167 squared degrees Celsius. std(ddof=0) uses the count itself and gives about 2.803 °C. NumPy's np.std defaults to ddof=0, so match the setting before comparing results.

Read quartiles and describe

quantile(0.25) gives the lower quartile, 14.5; quantile(0.5) gives the median, 16.0; quantile(0.75) gives the upper quartile, 18.5. The default interpolation can return a value between recorded readings.

weather.describe() combines count, mean, standard deviation, minimum, the three quartiles and maximum for numeric columns. Rain has mean about 2.94 mm and median 0.8 mm. The 12.3 mm reading pulls the mean upward; the median better describes a typical day in this right-skewed week.

Account for missing values and types

For pd.Series([1, np.nan, 3]), .mean() is 2.0 because missing values are skipped by default. .count() is 2, while .size is 3. With skipna=False, the mean is NaN. Do not use total row count as the denominator for the default mean when some entries are missing.

Text summaries differ. orders["category"].describe() gives count 6, unique 3, top Stationery and freq 3. Averaging the mixed-text orders table raises TypeError; use orders.mean(numeric_only=True) or select numeric columns with select_dtypes("number").

Summarize along an axis

By default, DataFrame statistics reduce down rows to one result per column. axis=1 instead gives one result per row; select quantities whose units can sensibly be combined.

orders[["quantity", "unit_price"]].agg(["mean", "max"])

agg computes several named statistics together. Quantity has mean 2.2 and maximum 4.0; unit_price has mean about 11.46 and maximum 35.0. weather["rain_mm"].cumsum() gives running totals 0.0, 2.4, 2.4, 2.4, 7.5, 19.8 and 20.6 mm, preserving one result per day.

Counts describe how often values occur; correlations describe how two numeric variables vary together. Both summaries need interpretation: what is being counted, what is missing and which relationship the statistic measures.

Count categories or list distinct values

orders["category"].value_counts() gives Stationery 3, Home 2 and Bags 1, ordered by frequency. The returned Series is named count. Missing entries are excluded by default; dropna=False includes them as a separate count.

orders["category"].value_counts(normalize=True)

The normalized Series is named proportion: Stationery is 0.5, Home about 0.333 and Bags about 0.167. Multiplying by 100 expresses percentages. With the default missing-value rule, these shares describe present values.

unique() keeps first-appearance order: Stationery, Home, Bags. mode() returns a Series because several values can tie for the highest frequency. It is not a request for a single arbitrarily chosen winner.

Conditions can be counted directly. (weather["rain_mm"] > 0).sum() counts four rainy days. Its .mean() is about 0.571, so about 57% of the seven days had rain.

Measure a straight-line relationship

weather["temp_c"].corr(weather["rain_mm"]) gives a Pearson correlation of about −0.397. Its negative sign says that warmer days tended to have less rain in these observations. Its magnitude describes how closely a straight-line pattern fits, on a scale from −1 through 0 to 1.

A value of 1 means a perfect increasing straight-line relationship; −1 means a perfect decreasing one. A value near zero means little linear association. A curved relationship can still be strong even when Pearson correlation is near zero. Correlation is also about association, not evidence that changing one variable causes the other to change.

Read a correlation matrix

weather.corr() gives all numeric pairs together. The matrix is symmetric: temperature with rain has the same correlation as rain with temperature. A varying numeric column has correlation 1 with itself; a constant column has no defined correlation and gives NaN instead.

For relationships that consistently increase or decrease without following a straight line, method="spearman" computes correlation from ranks. The method changes what kind of pattern is summarized, so name it when reporting a result.

The weather calculation uses only seven readings. It describes this small observed week; strong claims about a wider climate pattern need more evidence. A useful report includes the variables, the sign and size of the coefficient, the method and the number of observations.