PDA·03 · Chapter 3

Built-In Data Structures, Functions & Files

Chapter 3 — the containers, functions and file handling pandas is built on

Seven sessions on the parts of plain Python that data code leans on hardest: the four built-in containers, slicing, comprehensions, functions as values, generators, error handling, and reading and writing files.

3.1

Tuples

Concept

A tuple is a fixed-length, immutable sequence: tup = (4, 5, 6), or just 4, 5, 6 — the commas make the tuple, not the parentheses.
tuple() converts any sequence: tuple("string") gives ('s', 't', 'r', 'i', 'n', 'g').
The tuple can't change, but an object inside it can: if slot 1 holds a list, tup[1].append(3) works.
Unpacking assigns each element to a name: a, b, c = tup. It makes swapping one line: a, b = b, a.
*rest collects "everything else": a, b, *rest = values. By convention *_ means "values I don't need".

Key Terms

Tuple: an immutable, ordered sequence
Unpacking: assigning a sequence's elements to several names at once
*rest: captures the remaining elements as a list
count(): how many times a value appears

Worked example

values = 1, 2, 3, 4, 5
a, b, *rest = values
print(a, b, rest)

a, b = 1, 2
a, b = b, a
print(a, b)

seq = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]
for x, y, z in seq:
    print(f"x={x}, y={y}, z={z}")

1 2 [3, 4, 5]
2 1
x=1, y=2, z=3
x=4, y=5, z=6
x=7, y=8, z=9

Exercise

Activity 1

Store a student record as a tuple (name, class, score). Unpack it into three variables and print a sentence with an f-string.

Activity 2

Given scores = (55, 72, 91, 64, 88), use unpacking with *middle to get the first, last and everything in between.

3.2

Lists and Slicing

Concept

A list is variable-length and mutable: append, insert, pop, remove change it in place.
in checks membership, but on a list it scans every item — much slower than on a dict or set.
x.extend([...]) adds in place; x + [...] builds a whole new list. For building a big list, extend is faster.
sort() sorts in place; sort(key=len) sorts by a function of each item.
Slicing seq[start:stop] includes start and stops before stop, so it returns stop - start items. Omit either end to run to the edge; negative numbers count from the end.
A third number is the step: seq[::2] takes every other item, and seq[::-1] reverses the sequence.

Key Terms

append / insert: add at the end / at a position (insert is slower)
pop / remove: remove by position (and return it) / remove the first matching value
seq[start:stop:step]: a slice: start included, stop excluded, optional step
Negative index: counts from the end: -1 is the last item

Worked example

seq = [7, 2, 3, 7, 5, 6, 0, 1]
print(seq[1:5])
print(seq[-4:])
print(seq[::2])
print(seq[::-1])

b = ["saw", "small", "He", "foxes", "six"]
b.sort(key=len)
print(b)

[2, 3, 7, 5]
[5, 6, 0, 1]
[7, 3, 5, 0]
[1, 0, 6, 5, 7, 3, 2, 7]
['He', 'saw', 'six', 'small', 'foxes']

Exercise

Activity 1

Before using the visualiser, predict seq[2:], seq[:-2], seq[1::3] and seq[-1:-5:-1] on paper. Then check each one.

Activity 2

Find a slice that returns an empty list even though seq is not empty. Explain why it is empty.

Activity 3

Sort a list of words by their last letter using sort(key=...) and a lambda.

3.3

Dictionaries and Sets

Concept

A dict maps keys to values: d = {"a": "some value", "b": [1, 2, 3, 4]}. Look up with d[key]; d.get(key, default) avoids a KeyError.
keys(), values() and items() iterate in insertion order. update() merges another dict in place.
Building "lists by key" is so common there are helpers: setdefault(key, []).append(x), or collections.defaultdict(list).
Keys must be hashable — immutable things like strings, numbers and tuples of them. A list can't be a key; convert it to a tuple first.
A set holds unique items. | is union, & intersection, - difference, ^ symmetric difference.

Key Terms

dict: a mapping from hashable keys to values
get(key, default): look up without raising KeyError
defaultdict: a dict that creates a default value for missing keys
Hashable: immutable enough to be a dict key or set member
Set: an unordered collection of unique items

Worked example

from collections import defaultdict

words = ["apple", "bat", "bar", "atom", "book"]
by_letter = defaultdict(list)
for word in words:
    by_letter[word[0]].append(word)
print(dict(by_letter))

a = {1, 2, 3, 4, 5}
b = {3, 4, 5, 6, 7, 8}
print(a | b)
print(a & b)

{'a': ['apple', 'atom'], 'b': ['bat', 'bar', 'book']}
{1, 2, 3, 4, 5, 6, 7, 8}
{3, 4, 5}

Exercise

Activity 1

Group a list of students' names by the length of the name using setdefault, then again with defaultdict.

Activity 2

Two clubs have member lists with some overlap. Use sets to find who is in both, who is only in the first, and everyone in total.

3.4

Sequence Functions and Comprehensions

Concept

enumerate(seq) gives (index, value) pairs — use it instead of keeping your own counter.
sorted() returns a new sorted list from any sequence; reversed() iterates backwards; zip() pairs up several sequences, stopping at the shortest.
A list comprehension [expr for x in coll if cond] builds a filtered, transformed list in one line.
Set and dict comprehensions use braces: {len(x) for x in strings} and {val: i for i, val in enumerate(strings)}.
Nested comprehensions read in the same order as nested for loops: [x for tup in tuples for x in tup] flattens a list of tuples.

Key Terms

enumerate: yields (index, value) for each item
zip: pairs elements from several sequences
Comprehension: a concise expression that builds a list, set or dict

Worked example

strings = ["a", "as", "bat", "car", "dove", "python"]
print([x.upper() for x in strings if len(x) > 2])
print({len(x) for x in strings})
print({val: index for index, val in enumerate(strings)})

seq1 = ["foo", "bar", "baz"]
seq2 = ["one", "two", "three"]
for i, (a, b) in enumerate(zip(seq1, seq2)):
    print(f"{i}: {a}, {b}")

['BAT', 'CAR', 'DOVE', 'PYTHON']
{1, 2, 3, 4, 6}
{'a': 0, 'as': 1, 'bat': 2, 'car': 3, 'dove': 4, 'python': 5}
0: foo, one
1: bar, two
2: baz, three

Exercise

Activity 1

Given a list of names, build a dict of name → length with a dict comprehension, keeping only names with an "a" in them.

Activity 2

Flatten [(1, 2, 3), (4, 5, 6), (7, 8, 9)] into one list with a nested comprehension, then write the same thing as two for loops.

3.5

Functions

Concept

Functions are declared with def; arguments can be positional or keyword, and keyword arguments can have defaults.
Variables assigned inside a function live in its local namespace and vanish when it returns. Assigning to an outer variable needs global (rarely a good idea).
"Returning multiple values" is really returning one tuple, which the caller unpacks: a, b, c = f(). Returning a dict is often clearer.
Functions are objects: you can put them in a list and apply each in turn — a neat way to build a reusable cleaning pipeline.
A lambda is a small anonymous function in one expression, handy for key= arguments: strings.sort(key=lambda x: len(set(x))).

Key Terms

Keyword argument: an argument passed by name, often with a default
namespace / scope: where a variable name is visible
First-class function: a function that can be stored and passed like any value
lambda: an anonymous one-expression function

Worked example

import re

def remove_punctuation(value):
    return re.sub("[!#?]", "", value)

clean_ops = [str.strip, remove_punctuation, str.title]

def clean_strings(strings, ops):
    result = []
    for value in strings:
        for func in ops:
            value = func(value)
        result.append(value)
    return result

states = ["   Alabama ", "Georgia!", "georgia", "FlOrIda", "south   carolina##"]
print(clean_strings(states, clean_ops))

['Alabama', 'Georgia', 'Georgia', 'Florida', 'South   Carolina']

Exercise

Activity 1

Add a function to clean_ops that collapses repeated spaces, so "South Carolina" comes out with one space.

Activity 2

Write stats(numbers) that returns the minimum, maximum and mean, and unpack the result into three variables.

3.6

Generators and Errors

Concept

Anything you can loop over is iterable. A generator is a function that uses yield to produce values one at a time, lazily, only as they are asked for.
A generator expression is a comprehension in parentheses: sum(x ** 2 for x in range(100)) never builds the full list.
itertools has ready-made generators: groupby, combinations, permutations, product.
try/except handles errors. Catch specific exceptions — except ValueError: — so real bugs of other kinds still surface.
else runs only if the try succeeded; finally always runs, which is where cleanup like closing a file goes.

Key Terms

yield: hands back a value and pauses the generator until the next request
Lazy evaluation: computing values only when they are needed
itertools: standard module of iterator building blocks
try / except / else / finally: attempt, handle errors, run on success, always run

Worked example

def attempt_float(x):
    try:
        return float(x)
    except (TypeError, ValueError):
        return x

print(attempt_float("1.2345"))
print(attempt_float("something"))

import itertools
def first_letter(x):
    return x[0]

names = ["Alan", "Adam", "Wes", "Will", "Albert", "Steven"]
for letter, group in itertools.groupby(names, first_letter):
    print(letter, list(group))

1.2345
something
A ['Alan', 'Adam']
W ['Wes', 'Will']
A ['Albert']
S ['Steven']

Exercise

Activity 1

Notice "A" appears twice in the groupby output. Why? Fix it by sorting names by first letter first.

Activity 2

Write a generator squares(n) that yields 1, 4, 9 … up to n squared. Loop over it, then pass it to sum().

3.7

Files and the Operating System

Concept

open(path, encoding="utf-8") opens a file for reading in text mode. Always pass an encoding — the default differs between computers.
Use with open(...) as f: so the file is closed automatically, even if an error happens.
Modes: "r" read, "w" write (erases the file!), "x" write but fail if it exists, "a" append; add "b" for binary.
A text file iterates line by line. read(n) reads characters (text) or bytes (binary); tell() and seek() report and move the position.
In practice pandas does most file reading for you (Chapter 6), but this is what it is doing underneath.

Key Terms

with statement: a context manager that cleans up (closes the file) automatically
Text vs binary mode: str decoded with an encoding vs. raw bytes
Encoding: the rule for turning bytes into characters, such as UTF-8
rstrip(): removes trailing whitespace, including the newline

Worked example

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("first line\nsecond line\n")

with open("notes.txt", encoding="utf-8") as f:
    lines = [x.rstrip() for x in f]

print(lines)

['first line', 'second line']

Exercise

Activity 1

Write the multiplication table for 7 to a file, one line per row, then read it back and print only the even results.

Activity 2

Open the same file in "x" mode and read the error you get. When would that behaviour protect you?