Learn
Learning 🤖 AI-on 💻 On your computer ⏱ ~9h Difficulty 3/5 v1

Build a neural message classifier

A full-length text message is much more likely to be spam than a short one or a very long one. Your linear filter can only say "longer is more suspicious". Build the machinery behind every neural network (a small autograd engine and backpropagation), train a network with a hidden layer on the same four clues, and find out fairly what it gains.

Join free to start → Read the brief and materials. Join AI from Scratch to start, submit and ask the tutor. Log in

The problem: length is suspicious, but only up to a point

Here are real messages from the validation split, with the spam probability from the linear filter you trained in "Follow the gradient", and from the kind of network you will build here:

Length Message (start) Really Linear filter Network
485 "How to Make a girl Happy? It's not at all difficult to make girls happy. U only need to be... 1. A friend 2. C..." normal 74% 2%
431 "I can't keep going through this. It was never my intention to run you out, but if you choose to do that rather..." normal 58% 2%
155 "England v Macedonia - dont miss the goals/team news. Txt ur national team to 87077 eg ENGLAND to 87077..." spam 12% 63%
133 "PRIVATE! Your 2003 Account Statement for shows 800 un-redeemed S.I.M. points. Call 08715203685..." spam 8% 61%
143 "Every King Was Once A Crying Baby And Every Great Building Was Once A Map.. Not Imprtant Where U r TODAY..." normal 12% 63%

The linear filter flags two long, worried, personal messages, and misses two spam messages. The reason is the length clue. A text message holds about 160 characters, and spam senders fill it: messages of 130 to 165 characters are often spam, much shorter ones rarely, and very long ones (people writing several messages' worth) almost never. So suspicion should rise, then fall with length.

A linear filter cannot say that. Its score is w1*c1 + w2*c2 + w3*c3 + w4*length + b: one weight for length, so longer is always more suspicious (if w4 > 0) or always less. The network gets it right, and it pays a price too: look at the last row. Finding out honestly what a network gains, and what it costs, is the question of this quest.

To answer it, you build the machinery inside every neural network: numbers that remember how they were computed, and backpropagation, which finds all the gradients at once. In "Follow the gradient" you derived the gradient by hand for one formula. Here the computer does it for any formula you build.

What you'll build

File What it does
engine.py Value: a number that remembers how it was made, and backward() (the grader tests this)
nn.py Neuron, Layer, MLP, with save_weights and load_weights (the grader tests this)
train.py your experiments: the length table, training, the comparison (anyone can rerun it)
mlp_weights.json your trained network, written by save_weights
SUBMISSION.md your write-up (the starter kit has a template)

Your starter kit also has, marked "Provided", the clues (features.py), the filter and loss from the earlier quests (filter.py, learn.py) and the linear filter's training (gradients.py). You may replace them with your own. Pure Python only: math, random and json, no other libraries.

Step 1: look at the data first

Before you build anything, check the claim above yourself. Group the training messages by length (0-50, 50-100, 100-130, 130-165, 165-250 and over 250 characters) and count the share of spam in each group. A small table or a plot is enough. Which group is the most suspicious? Is there any spam above 250 characters?

The ideas, with small numbers

1. Neurons and a hidden layer make a bump

A neuron computes a score like your filter, then squashes it with an activation function. With tanh, the output is between -1 and 1:

neuron(x) = tanh(w1*x1 + w2*x2 + ... + b)

Two neurons that both read only the length L can make a bump. Take h1 = tanh(10 * (L - 1.3)) and h2 = tanh(10 * (L - 1.65)), and use h1 - h2:

Length L h1 h2 h1 - h2
100 characters 1.0 -0.995 -1.000 0.005
150 characters 1.5 0.964 -0.905 1.869
300 characters 3.0 1.000 1.000 0.000

High only in the middle: rise, then fall. A layer is several neurons reading the same inputs; a hidden layer is one whose outputs feed another layer instead of being the answer. Your network, MLP(4, [8, 1]), has 4 inputs (the clues), a hidden layer of 8 tanh neurons, and one linear output neuron that combines them into the spam score. Nobody tells it to build a bump: training finds whatever the data needs. (Without the activation, two layers would collapse into one linear score. Try to see why.)

2. A computation graph remembers how a number was made

a = Value(2.0); b = Value(-3.0)
c = a * b + a        # c = -4
c.backward()
print(a.grad, b.grad)    # -2.0 and 2.0

Each Value remembers the values it was made from and the operation. backward() walks this computation graph from the output back to the inputs and applies the chain rule from the last quest at every step. Each operation only needs its own local slope: for x * y, the slope with respect to x is y.

Check the numbers: c = a*b + a, so a small change in a changes c through two paths: through a*b (slope b = -3) and directly (slope 1). Total: -3 + 1 = -2. When a value is used in several places, its gradient is the sum over all the places. This is the most common bug in autograd engines: writing grad = ... instead of grad += ....

3. Standardize the inputs

The clues have very different sizes: caps_fraction is between 0 and 1, length goes up to 8, prize_words is often 0. Training is much faster when every input is centred and scaled: (x - mean) / std. For length, the training mean is about 0.81 and the standard deviation about 0.59, so a 150-character message becomes (1.5 - 0.81) / 0.59 = 1.17. Compute the means and standard deviations from the training split only, and save them with the weights (input_mean, input_std), because every future message must be scaled the same way.

4. Training on small batches

In the last quest each step used all 3,900 messages. With a network in pure Python that is too slow, so each step uses a small random batch (for example 32 messages): stochastic gradient descent. The steps are noisier but much cheaper. Plain updates, p.data -= learning_rate * p.grad, are enough; you don't need a fancier optimizer. Reset every gradient to 0 before each backward().

Your tasks

  1. The length table (Step 1 above).
  2. The engine. Write Value in engine.py. Start with + and *, check a * b + a by hand, then add the other operations. Check each one with finite differences (nudge an input, as in the last quest), including a graph where a value is used more than once.
  3. The network. Write Neuron, Layer and MLP in nn.py, with save_weights and load_weights in the format below. Check one gradient of a small network against finite differences.
  4. Train it. Train MLP(4, [8, 1]) (or another small size you can justify) with cross-entropy on the training split. Use the validation split to choose settings. Save mlp_weights.json.
  5. Compare fairly with the linear filter: same training data, same loss, same validation messages, same rule (spam when the probability is above 0.5). A neat way: MLP(4, [1]) is a linear filter, so you can train both with the same code; or use gradients.py. Report loss, recall, precision and accuracy for both. Then look inside: the spam probability as the length changes with the other clues fixed, and a few messages the two models disagree on.

The weights file

save_weights(path) writes, and load_weights(path) reads, this JSON:

{
  "n_inputs": 4,
  "layer_sizes": [8, 1],
  "activation": "tanh",
  "input_mean": [0.03, 0.25, 0.09, 0.81],
  "input_std": [0.17, 0.90, 0.15, 0.59],
  "layers": [
    {"weights": [[w, w, w, w], ... 8 rows, one per hidden neuron], "biases": [b, ... 8 numbers]},
    {"weights": [[w, w, w, w, w, w, w, w]], "biases": [b]}
  ]
}

Row k of a layer's "weights" holds neuron k's weights, one per input of that layer. The network first standardizes the clues with input_mean and input_std, runs the hidden layers with the activation, and the last layer is linear. Its single output is the spam score; the spam probability is sigmoid(score).

Rules the grader checks

  • Value supports + - * /, -x, x ** n (n a plain number), exp, log, tanh, relu and sigmoid, with a plain number on either side (2 - x and x - 2 both work).
  • backward() sets the output's .grad to 1 and fills .grad for every value in the graph. A value used in several places gets the sum of the gradients. Graphs can be a few hundred operations deep.
  • relu has gradient 0 at exactly 0. tanh(1000) is 1 and sigmoid(-1000) is 0, without errors. sigmoid is one operation with its own local slope.
  • MLP(n_inputs, layer_sizes, activation="tanh"): hidden layers use the activation ("tanh" or "relu"), the last layer is linear, one output gives one Value. parameters() returns each parameter once. Neuron(4) has 5 parameters; MLP(4, [8, 1]) has 49.
  • save_weights and load_weights follow the format above exactly. The grader reads your mlp_weights.json with its own code.

The exact interface is in the "Interface" section at the end of this brief.

The core pieces you must be able to explain

You may use AI for anything in this quest. The defense then checks that you understand your own submission. Be ready to:

  • trace backward() by hand on a graph of three or four operations, including a reused value;
  • explain why gradients add up (+=) and why the nodes are visited in reverse order;
  • give the local slope of any operation in your engine, for example sigmoid or tanh;
  • count the parameters of your network and say what the hidden layer does with the length clue;
  • explain why a linear filter cannot make "rise, then fall", and why stacking linear layers without an activation does not help;
  • defend your comparison: what was kept the same, which numbers changed, what accuracy hides;
  • change your code: for example, add a new operation or a second hidden layer.

How you're graded

  1. Automated checks, on a public suite (you have it; neugen test runs it) and a hidden one: - Gradients on expression graphs and Network checks: all must pass. The hidden suite adds deep graphs, values used many times, edge cases, relu networks and two hidden layers. - Your network, scored on the validation split (public) and on hidden test messages: loss at most 0.16 (no linear filter can get there), recall at least 0.68, precision at least 0.7. - Rise, then fall: your network must rate a 150-character message as more suspicious than a 50- or a 300-character one with the same other clues (no link, one prize word, 10% capitals), by at least 0.1 in probability. A linear filter always fails this.
  2. Write-up review of SUBMISSION.md with the rubric on this page.
  3. Defense: a short conversation with the AI examiner about your code and results.

Suggested path (about 9 hours)

  1. Read the CS231n sections in the materials (1.5 hours). Trace a * b + a on paper.
  2. The length table (30 minutes).
  3. engine.py, with finite-difference checks as you go (2.5 hours).
  4. nn.py and the weights file (1.5 hours).
  5. Training and choosing settings (1.5 hours).
  6. The comparison and a look inside (1 hour).
  7. Write-up, neugen test, submit (30 minutes).

Tips

  • Check every operation with finite differences the moment you write it. A bug in one local slope makes every network gradient wrong, and training then fails in confusing ways.
  • Speed. Work out the clues once. A few thousand steps of 32 messages take a minute or two in pure Python for a 4-8-1 network; much bigger networks get slow.
  • If training is stuck at the linear filter's loss (about 0.18), check that the inputs are standardized and that the gradients are reset each step, then try a learning rate 10 times larger or smaller. Print the validation loss every few hundred steps.
  • Seeds. Starting weights and batches are random. Seed them so your numbers can be reproduced, and try more than one seed before you draw a conclusion.
  • Accuracy hides the gain. The two models have almost the same accuracy, because 86% of messages are normal and most are easy. Look at loss and recall, and at the messages that changed.
  • A bigger network is not automatically better. If the training loss keeps falling while the validation loss rises, the network is learning the training messages, not the pattern.

Interface

# engine.py
class Value:
    data: float; grad: float            # grad starts at 0.0
    def __init__(self, data): ...
    # a + b, a - b, a * b, a / b, -a, a ** n (n a plain int or float), and the same with a
    # plain number on either side: 2 + a, 2 - a, 2 * a, 2 / a
    def exp(self) -> "Value": ...
    def log(self) -> "Value": ...
    def tanh(self) -> "Value": ...
    def relu(self) -> "Value": ...      # gradient 0 at exactly 0
    def sigmoid(self) -> "Value": ...   # one operation; sigmoid(-1000) == 0.0, no overflow
    def backward(self) -> None: ...     # fills .grad of every node in the graph

# nn.py  (activation is "tanh" or "relu"; the last layer is linear)
class Neuron: def __init__(self, n_inputs, activation="tanh"): ...   # __call__(x) -> Value
class Layer:  def __init__(self, n_inputs, n_outputs, activation="tanh"): ...   # -> list
class MLP:
    def __init__(self, n_inputs, layer_sizes, activation="tanh"): ...
    input_mean: list[float]; input_std: list[float]   # inputs become (x - mean) / std
    def __call__(self, x): ...          # one output -> one Value (the spam score)
    def parameters(self) -> list: ...   # each Value once
    def save_weights(self, path): ...   # the JSON format in nn.py and the brief
    def load_weights(self, path): ...   # into this network (same sizes and activation)
# each of Neuron, Layer, MLP has parameters()

# mlp_weights.json: your trained network, written by save_weights.
# A message is flagged as spam when its score is above 0 (probability above 0.5).