Machine learning approaches

A4.3.1 Linear regression

Linear regression finds the straight line of best fit, then uses it to predict a continuous value. It is only suitable when the relationship really is roughly linear, so check a scatter plot first.

y = a + bx   ·   b = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)²   ·   a = ȳ − bx̄

The least squares line minimizes the sum of the squared vertical distances to the points. R², the coefficient of determination, shows how much of the variation the line explains: near 1 is a good fit, and near 0 a poor one.

import numpy as np
from sklearn.linear_model import LinearRegression

hours = np.array([1, 2, 3, 4, 5, 6]).reshape(-1, 1)   # one column
marks = np.array([35, 48, 52, 66, 71, 83])

model = LinearRegression().fit(hours, marks)
print(model.intercept_, model.coef_[0])
print(model.predict([[7]]))
print(model.score(hours, marks))                      # R squared

With several inputs this becomes multiple linear regression, which fits a plane rather than a line. Other measures of error include mean squared error and mean absolute error.

Next: classification →

A4.3.2 Classification

Classification predicts a category. Examples include spam filters, medical diagnosis, credit risk and sentiment analysis.

K-nearest neighbours (KNN)

To classify a new point, find the k closest labelled points and let them vote. KNN is a lazy learner, since it does almost nothing in training and does the work at prediction time.

• Scale the data first, as KNN depends on distances.
• With two classes, use an odd k to avoid ties.
• A k of 1 is easily fooled by outliers.

Decision trees

A tree is a learned flowchart of questions. Trees are eager learners, doing the work in training to choose the questions and thresholds.

Gini impurity at a node measures how mixed the classes are, where 0 means every sample has the same class. An even three-way mix scores 0.667.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

x, y = load_iris(return_X_y=True)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(x_train, y_train)
print(tree.score(x_test, y_test))

Next: evaluation and tuning →

A4.3.3 Evaluation and tuning

• Use precision when a false positive is costly, such as a real email marked as spam.
• Use recall when a false negative is costly, such as a missed disease.
• Use F1 when both matter. It is a harmonic mean, so it stays close to the smaller value.

Hyperparameters

A hyperparameter is set before learning and guides it, such as k, tree depth, the learning rate or the number of hidden layers. Tuning means trying combinations and comparing them on validation data.

Next: clustering →

A4.3.4 Clustering

Clustering is unsupervised: there are no labels, so the algorithm groups points by similarity.

Next: association rules →

A4.3.5 Association rules

Association rule learning finds items that occur together. It is another unsupervised technique, used in market analysis, healthcare and tracking app behaviour.

support(A→B) = both ÷ all   ·   confidence = both ÷ A   ·   lift = confidence ÷ support(B)

The Apriori algorithm sets a minimum support, keeps the single items that reach it, then builds larger item sets. Any set that fails can be dropped with all its supersets, which saves a lot of work.

Next: reinforcement learning →

A4.3.6 Reinforcement learning

An agent acts in an environment, observes the state, receives a reward, and improves its policy, the mapping from states to actions.

• Exploration: try something new.
• Exploitation: use what has been learned.

Q-learning

A Q-table stores the expected value of each action in each state. If there are too many states for a table, a neural network can approximate it, which is called a Deep Q-Network.

Q(S, A) ← Q(S, A) + alpha × (R + gamma × max Q(S', all actions) − Q(S, A))

Next: genetic algorithms →

A4.3.7 Genetic algorithms

A genetic algorithm is an optimization method inspired by natural selection. It suits problems where a good answer will do and the perfect answer is out of reach.

1. Population: start with many random candidate solutions.
2. Fitness function: score each one.
3. Selection: pick parents, with fitter ones more likely. A roulette wheel gives each a share based on its fitness. Keeping some randomness helps avoid local optima.
4. Crossover: combine parts of two parents.
5. Mutation: randomly change a small share of children.
6. Repeat until a set number of generations, or a target score.

Uses include route planning, timetabling, engineering design, robot controllers and choosing features for machine learning.

Next: neural networks →

A4.3.8 Artificial neural networks

An ANN has an input layer with one neuron per feature, one or more hidden layers that find patterns, and an output layer that gives the prediction. In a fully connected network, each neuron feeds every neuron in the next layer.

output = activation( Σ(input × weight) + bias )

Activation functions

Activation adds non-linearity. Without it, any number of layers would behave like a single linear function.

Training by backpropagation

1. Forward pass: compute a prediction.
2. Loss: measure the error, with mean squared error for regression or cross-entropy for classification.
3. Backpropagation: work back through the layers to find each weight’s contribution to the error.
4. Gradient descent: adjust weights to reduce loss, scaled by the learning rate.
5. Repeat for many epochs, or full passes over the data.

Next: CNNs →

A4.3.9 Convolutional neural networks

Even a 100 × 100 colour image holds 300,000 values, which is too many for a plain ANN. A CNN first extracts features and shrinks the data. Early layers find simple edges, and later layers combine them into shapes, building a spatial hierarchy of features.

1. Input: raw pixels, sized width × height × colour channels.
2. Convolution: filters (kernels) slide over the image. Their values are learned in training.
3. Activation: usually ReLU, adding non-linearity.
4. Pooling: max or average pooling shrinks the feature maps, reducing cost and sensitivity to noise.
5. Fully connected: data is flattened and passed to an ANN, which classifies it.
6. Output: the final prediction.

from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Input(shape=(32, 32, 3)),
    layers.Conv2D(32, (3, 3), activation="relu"),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation="relu"),
    layers.MaxPooling2D((2, 2)),
    layers.Flatten(),
    layers.Dense(64, activation="relu"),
    layers.Dense(10, activation="softmax"),
])

Next: model selection →

A4.3.10 Model selection

No model is best for everything. Start simple to set a baseline, then add complexity only if it helps.

Cross-validation splits the data into several folds, training on some and testing on the rest in rotation. It gives a fairer comparison than a single split, and the exam may ask about it.

Next: review →

Quick quiz

Exam-style questions

These are written for this site in the style of Paper 1. Answer on paper first, then open the mark points.

1. A shop uses linear regression to predict spending. State its assumption, and describe how an outlier could affect it. [3]

   Mark points

   • It assumes a linear relationship between the independent and dependent variables [1].
   • An outlier pulls the line of best fit towards itself [1].
   • This changes the slope, making predictions for typical customers worse [1].

2. An online retailer uses KNN to classify reviews. Outline how the choice of k affects results, and describe how to find a good k. [4]

   Mark points

   • A small k is sensitive to noise and outliers [1].
   • A large k smooths the boundary but can underfit, ignoring local patterns [1].
   • Try a range of k values on validation data [1].
   • Choose the k with the best score, such as accuracy or F1 [1].

3. A plant predicts equipment failure. Define precision and recall here, and explain why F1 may be better than accuracy. [4]

   Mark points

   • Precision: of predicted failures, the proportion that really failed [1].
   • Recall: of actual failures, the proportion predicted [1].
   • Failures are rare, so always predicting “no failure” would score high accuracy [1].
   • F1 combines both, so it exposes this weakness [1].

4. Outline how k-means clusters customers. [3]

   Mark points

   • Choose k and place k starting centres [1].
   • Assign each customer to the nearest centre [1].
   • Move each centre to the mean of its points, and repeat until they settle [1].

5. Define support, confidence and lift for a library’s borrowing data. [3]

   Mark points

   • Support: the proportion of loans containing both genres [1].
   • Confidence: of loans with genre A, the proportion that also have genre B [1].
   • Lift: confidence divided by the overall proportion with B, showing strength beyond chance [1].

6. A school uses a genetic algorithm to allocate electives. Describe a suitable fitness function, and outline selection and crossover. [4]

   Mark points

   • Fitness: score each timetable by how many students get a first or second choice [1].
   • Timetables that break a constraint, such as room limits, score lower [1].
   • Selection: fitter timetables are more likely to be chosen as parents [1].
   • Crossover: two parents are combined to create a new timetable [1].

7. An energy company forecasts electricity demand using a neural network. Identify the problem type and suitable output layer, and define backpropagation. [4]

   Mark points

   • Regression, as demand is a continuous number [1].
   • One output neuron with a linear activation [1].
   • Backpropagation calculates how much each weight contributed to the error, working back from the output [1].
   • Weights are adjusted in the direction that reduces loss [1].

8. Describe the role of convolutional and pooling layers in a CNN. [4]

   Mark points

   • Convolution slides filters over the image [1].
   • This produces feature maps that show edges and textures [1].
   • Pooling reduces the dimensions of the maps [1].
   • This lowers computation and sensitivity to small changes [1].

9. Outline cross-validation and why it matters. [3]

   Mark points

   • Split the data into several folds [1].
   • Train on some folds and test on the remaining fold, rotating through all of them [1].
   • This gives a more reliable estimate of performance on unseen data [1].