PDA·05 · Chapter 5

Getting Started with pandas

Chapter 5 — Series, DataFrame, and the essentials of working with them

This is where the book becomes about pandas. Six sessions: the two core data structures, reshaping their labels, the selection rules that trip everyone up (loc vs. iloc), how arithmetic lines up by label, and the one-line summaries you'll use on every dataset.

5.1

Series

Concept

A Series is a one-dimensional array of values plus an index — a label for each value. Without one, pandas numbers them 0, 1, 2…
Select by label (obj["a"]) or with a boolean mask (obj[obj > 0]). NumPy-style maths keeps the labels attached to the values.
A Series is like an ordered dict: build one from a dict, and pass index= to choose and order the keys. A key with no value becomes NaN, pandas' marker for missing data.
isna() and notna() find the missing values.
The big idea: arithmetic aligns on labels, not positions. obj3 + obj4 adds Ohio to Ohio and Texas to Texas, whatever order they are in; labels found in only one side give NaN.

Key Terms

Series: a 1-D labelled array
Index: the labels for each position
NaN: "not a number" — pandas' standard marker for missing values
Alignment: matching values by label before doing arithmetic

Worked example

import pandas as pd

sdata = {"Ohio": 35000, "Texas": 71000, "Oregon": 16000, "Utah": 5000}
states = ["California", "Ohio", "Oregon", "Texas"]
obj3 = pd.Series(sdata)
obj4 = pd.Series(sdata, index=states)   # no value for California
print(obj4)
print(obj3 + obj4)                      # aligned by state name

California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64
California         NaN
Ohio           70000.0
Oregon         32000.0
Texas         142000.0
Utah               NaN
dtype: float64

Exercise

Activity 1

Make a Series of five classmates' heights with their names as the index. Select everyone taller than the mean.

Activity 2

Make a second Series of weights for four of the same people plus one new person, in a different order. Add the two Series together and explain every NaN.

5.2

DataFrame and Index Objects

Concept

A DataFrame is a table: an ordered collection of columns, each of which can have a different type. It has an index for rows and an index for columns.
The most common way to build one is a dict of equal-length lists. head() and tail() show the first or last rows.
frame["state"] gets one column as a Series. Assigning to a new column name adds it: frame["eastern"] = frame["state"] == "Ohio". del frame["eastern"] removes it.
frame.T swaps rows and columns; frame.dtypes shows each column's type. New in pandas 3: text columns now show str, not object.
Index objects hold the labels. They are immutable, so they can be safely shared between Series and DataFrames.

Key Terms

DataFrame: a 2-D labelled table of columns
Column: a Series inside a DataFrame, sharing its row index
head() / tail(): first / last rows (5 by default)
dtypes: the data type of every column

Worked example

data = {"state": ["Ohio", "Ohio", "Ohio", "Nevada", "Nevada", "Nevada"],
        "year": [2000, 2001, 2002, 2001, 2002, 2003],
        "pop": [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)
frame["eastern"] = frame["state"] == "Ohio"
print(frame)
print(frame.dtypes)

    state  year  pop  eastern
0    Ohio  2000  1.5     True
1    Ohio  2001  1.7     True
2    Ohio  2002  3.6     True
3  Nevada  2001  2.4    False
4  Nevada  2002  2.9    False
5  Nevada  2003  3.2    False
state          str
year         int64
pop        float64
eastern       bool
dtype: object

Exercise

Activity 1

Build a DataFrame of five books with title, author, year and pages. Add a column "long" that is True when pages > 300.

Activity 2

Print frame.T for your books table. When would a transposed table be easier to read?

5.3

Reindexing and Dropping

Concept

reindex makes a new object that follows a new index: existing labels keep their values, new labels get NaN, missing labels are dropped.
For ordered data like time series, method="ffill" fills each gap with the previous value.
drop returns a new object without the given labels: drop(index=[...]) for rows, drop(columns=[...]) for columns. The original is unchanged.
Most pandas methods return a new object rather than changing the one you have. Assign the result back if you want to keep it: df = df.drop(columns="two").

Key Terms

reindex: conform data to a new set of labels
ffill: forward fill: carry the last value forward
drop: remove rows or columns by label, returning a new object

Worked example

import numpy as np

obj = pd.Series([4.5, 7.2, -5.3, 3.6], index=["d", "b", "a", "c"])
print(obj.reindex(["a", "b", "c", "d", "e"]))

data = pd.DataFrame(np.arange(16).reshape((4, 4)),
                    index=["Ohio", "Colorado", "Utah", "New York"],
                    columns=["one", "two", "three", "four"])
print(data.drop(index=["Colorado", "Ohio"]))

a   -5.3
b    7.2
c    3.6
d    4.5
e    NaN
dtype: float64
          one  two  three  four
Utah        8    9     10    11
New York   12   13     14    15

Exercise

Activity 1

Reindex the books table so its rows come out in reverse order, then drop the "pages" column. Check the original still has it.

Activity 2

Make pd.Series(["blue", "purple", "yellow"], index=[0, 2, 4]) and reindex it to range(6) with and without method="ffill".

5.4

Selecting with loc and iloc

Concept

On a DataFrame, plain [] selects columns: data["two"] or data[["three", "one"]]. A boolean Series inside [] filters rows.
loc selects by label; iloc selects by integer position. Both take [rows, columns].
Label slices include the end: data.loc[:"Utah"] includes Utah. Position slices don't: data.iloc[:2] is rows 0 and 1.
Pitfall: on a Series with an integer index, ser[-1] looks for the label −1 and raises KeyError. Say what you mean: ser.iloc[-1].
Pitfall: data[data["three"] > 5]["three"] = 0 is chained assignment — the first [] makes a new object, so the change is lost. In pandas 3 it never works and warns. Do it in one step: data.loc[data["three"] > 5, "three"] = 0.

Key Terms

df[col]: select column(s) by name
loc[rows, cols]: select by label; label slices include the end
iloc[rows, cols]: select by position; slices exclude the end
Chained assignment: df[a][b] = x — sets a value on a temporary copy

Worked example

print(data.loc["Colorado", ["two", "three"]])
print(data.iloc[[2, 1], [3, 0, 1]])

data.loc[data["three"] > 5, "three"] = 0   # one step, not data[...][...]
print(data["three"].tolist())

two      5
three    6
Name: Colorado, dtype: int64
          four  one  two
Utah        11    8    9
Colorado     7    4    5
[2, 0, 0, 0]

Exercise

Activity 1

Use the selector below. For each expression, predict which cells light up and whether the result is a Series, a DataFrame or a single value.

Activity 2

Write two ways to get the "four" column for Colorado and Utah: once with loc, once with iloc.

Activity 3

Try the three error buttons. Explain in one sentence each why they fail.

5.5

Arithmetic, Functions and Sorting

Concept

Adding two objects aligns their labels: the result has the union of both indexes, with NaN wherever a label is missing from either side.
The method versions take a fill_value for the missing side: s1.add(s2, fill_value=0). Also sub, mul, div.
NumPy ufuncs work on whole DataFrames. frame.apply(f) calls f once per column (or per row with axis="columns").
For element-wise functions use DataFrame.map(f); the book's applymap was renamed in pandas 2.1. On a Series it is Series.map.
sort_index() sorts by label; sort_values("col") sorts by values, with NaN placed at the end. rank() gives each value its rank.

Key Terms

fill_value: value used for a label missing from one side
apply: run a function on each column or row
DataFrame.map: run a function on every element (was applymap)
sort_values / sort_index: sort by values / by labels

Worked example

s1 = pd.Series([7.3, -2.5, 3.4, 1.5], index=["a", "c", "d", "e"])
s2 = pd.Series([-2.1, 3.6, -1.5, 4, 3.1], index=["a", "c", "e", "f", "g"])
print(s1 + s2)
print(s1.add(s2, fill_value=0))

a    5.2
c    1.1
d    NaN
e    0.0
f    NaN
g    NaN
dtype: float64
a    5.2
c    1.1
d    3.4
e    0.0
f    4.0
g    3.1
dtype: float64

Exercise

Activity 1

Make two Series of weekly sales for two shops with partly different product names. Add them with + and with add(fill_value=0). Which answer is right for "total sales"?

Activity 2

On the books table, use apply to get max − min of every numeric column, then sort the books by year, newest first.

5.6

Descriptive Statistics

Concept

Reductions like sum, mean, min, max work per column by default, and skip missing values unless you pass skipna=False.
idxmax() / idxmin() return the label where the max/min is; cumsum() gives running totals.
describe() gives count, mean, std, min, quartiles and max for every numeric column in one go.
corr() and cov() measure how columns move together — e.g. the daily returns of two stocks.
For categories: unique() lists distinct values, value_counts() counts them (most common first), and isin([...]) gives a mask for filtering.

Key Terms

skipna: whether reductions ignore NaN (default True)
idxmax: label of the maximum value
describe: summary statistics for each column
value_counts: frequency of each distinct value

Worked example

df = pd.DataFrame([[1.4, np.nan], [7.1, -4.5], [np.nan, np.nan], [0.75, -1.3]],
                  index=["a", "b", "c", "d"], columns=["one", "two"])
print(df.sum())
print(df.idxmax())

obj = pd.Series(["c", "a", "d", "a", "a", "b", "b", "c", "c"])
print(obj.value_counts())

one    9.25
two   -5.80
dtype: float64
one    b
two    d
dtype: str
c    3
a    3
b    2
d    1
Name: count, dtype: int64

Exercise

Activity 1

Run describe() on the books table. Which numbers are useful for the "year" column, and which are meaningless?

Activity 2

Make a Series of 20 favourite colours from your class. Find the top three with value_counts and filter to only red or blue with isin.