Build a neural message classifier
A full-length text message is much more likely to be spam than a short one or a very long one. Your linear filter can only say "longer is more suspicious". Build the machinery behind every neural network (a small autograd engine and backpropagation), train a network with a hidden layer on the same four clues, and find out fairly what it gains.
The problem: length is suspicious, but only up to a point
Here are real messages from the validation split, with the spam probability from the linear filter you trained in "Follow the gradient", and from the kind of network you will build here:
| Length | Message (start) | Really | Linear filter | Network |
|---|---|---|---|---|
| 485 | "How to Make a girl Happy? It's not at all difficult to make girls happy. U only need to be... 1. A friend 2. C..." | normal | 74% | 2% |
| 431 | "I can't keep going through this. It was never my intention to run you out, but if you choose to do that rather..." | normal | 58% | 2% |
| 155 | "England v Macedonia - dont miss the goals/team news. Txt ur national team to 87077 eg ENGLAND to 87077..." | spam | 12% | 63% |
| 133 | "PRIVATE! Your 2003 Account Statement for shows 800 un-redeemed S.I.M. points. Call 08715203685..." | spam | 8% | 61% |
| 143 | "Every King Was Once A Crying Baby And Every Great Building Was Once A Map.. Not Imprtant Where U r TODAY..." | normal | 12% | 63% |
The linear filter flags two long, worried, personal messages, and misses two spam messages. The
reason is the length clue. A text message holds about 160 characters, and spam senders fill it:
messages of 130 to 165 characters are often spam, much shorter ones rarely, and very long ones
(people writing several messages' worth) almost never. So suspicion should rise, then fall
with length.
A linear filter cannot say that. Its score is w1*c1 + w2*c2 + w3*c3 + w4*length + b: one weight
for length, so longer is always more suspicious (if w4 > 0) or always less. The network gets
it right, and it pays a price too: look at the last row. Finding out honestly what a network
gains, and what it costs, is the question of this quest.
To answer it, you build the machinery inside every neural network: numbers that remember how they were computed, and backpropagation, which finds all the gradients at once. In "Follow the gradient" you derived the gradient by hand for one formula. Here the computer does it for any formula you build.
What you'll build
| File | What it does |
|---|---|
engine.py |
Value: a number that remembers how it was made, and backward() (the grader tests this) |
nn.py |
Neuron, Layer, MLP, with save_weights and load_weights (the grader tests this) |
train.py |
your experiments: the length table, training, the comparison (anyone can rerun it) |
mlp_weights.json |
your trained network, written by save_weights |
SUBMISSION.md |
your write-up (the starter kit has a template) |
Your starter kit also has, marked "Provided", the clues (features.py), the filter and loss from
the earlier quests (filter.py, learn.py) and the linear filter's training (gradients.py). You
may replace them with your own. Pure Python only: math, random and json, no other libraries.
Step 1: look at the data first
Before you build anything, check the claim above yourself. Group the training messages by length (0-50, 50-100, 100-130, 130-165, 165-250 and over 250 characters) and count the share of spam in each group. A small table or a plot is enough. Which group is the most suspicious? Is there any spam above 250 characters?
The ideas, with small numbers
1. Neurons and a hidden layer make a bump
A neuron computes a score like your filter, then squashes it with an activation function.
With tanh, the output is between -1 and 1:
neuron(x) = tanh(w1*x1 + w2*x2 + ... + b)
Two neurons that both read only the length L can make a bump. Take
h1 = tanh(10 * (L - 1.3)) and h2 = tanh(10 * (L - 1.65)), and use h1 - h2:
| Length | L | h1 | h2 | h1 - h2 |
|---|---|---|---|---|
| 100 characters | 1.0 | -0.995 | -1.000 | 0.005 |
| 150 characters | 1.5 | 0.964 | -0.905 | 1.869 |
| 300 characters | 3.0 | 1.000 | 1.000 | 0.000 |
High only in the middle: rise, then fall. A layer is several neurons reading the same inputs;
a hidden layer is one whose outputs feed another layer instead of being the answer. Your
network, MLP(4, [8, 1]), has 4 inputs (the clues), a hidden layer of 8 tanh neurons, and one
linear output neuron that combines them into the spam score. Nobody tells it to build a bump:
training finds whatever the data needs. (Without the activation, two layers would collapse into
one linear score. Try to see why.)
2. A computation graph remembers how a number was made
a = Value(2.0); b = Value(-3.0)
c = a * b + a # c = -4
c.backward()
print(a.grad, b.grad) # -2.0 and 2.0
Each Value remembers the values it was made from and the operation. backward() walks this
computation graph from the output back to the inputs and applies the chain rule from the
last quest at every step. Each operation only needs its own local slope: for x * y, the slope
with respect to x is y.
Check the numbers: c = a*b + a, so a small change in a changes c through two paths: through
a*b (slope b = -3) and directly (slope 1). Total: -3 + 1 = -2. When a value is used in
several places, its gradient is the sum over all the places. This is the most common bug in
autograd engines: writing grad = ... instead of grad += ....
3. Standardize the inputs
The clues have very different sizes: caps_fraction is between 0 and 1, length goes up to 8,
prize_words is often 0. Training is much faster when every input is centred and scaled:
(x - mean) / std. For length, the training mean is about 0.81 and the standard deviation about
0.59, so a 150-character message becomes (1.5 - 0.81) / 0.59 = 1.17. Compute the means and
standard deviations from the training split only, and save them with the weights
(input_mean, input_std), because every future message must be scaled the same way.
4. Training on small batches
In the last quest each step used all 3,900 messages. With a network in pure Python that is too
slow, so each step uses a small random batch (for example 32 messages): stochastic gradient
descent. The steps are noisier but much cheaper. Plain updates, p.data -= learning_rate *
p.grad, are enough; you don't need a fancier optimizer. Reset every gradient to 0 before each
backward().
Your tasks
- The length table (Step 1 above).
- The engine. Write
Valueinengine.py. Start with+and*, checka * b + aby hand, then add the other operations. Check each one with finite differences (nudge an input, as in the last quest), including a graph where a value is used more than once. - The network. Write
Neuron,LayerandMLPinnn.py, withsave_weightsandload_weightsin the format below. Check one gradient of a small network against finite differences. - Train it. Train
MLP(4, [8, 1])(or another small size you can justify) with cross-entropy on the training split. Use the validation split to choose settings. Savemlp_weights.json. - Compare fairly with the linear filter: same training data, same loss, same validation
messages, same rule (spam when the probability is above 0.5). A neat way:
MLP(4, [1])is a linear filter, so you can train both with the same code; or usegradients.py. Report loss, recall, precision and accuracy for both. Then look inside: the spam probability as the length changes with the other clues fixed, and a few messages the two models disagree on.
The weights file
save_weights(path) writes, and load_weights(path) reads, this JSON:
{
"n_inputs": 4,
"layer_sizes": [8, 1],
"activation": "tanh",
"input_mean": [0.03, 0.25, 0.09, 0.81],
"input_std": [0.17, 0.90, 0.15, 0.59],
"layers": [
{"weights": [[w, w, w, w], ... 8 rows, one per hidden neuron], "biases": [b, ... 8 numbers]},
{"weights": [[w, w, w, w, w, w, w, w]], "biases": [b]}
]
}
Row k of a layer's "weights" holds neuron k's weights, one per input of that layer. The network
first standardizes the clues with input_mean and input_std, runs the hidden layers with the
activation, and the last layer is linear. Its single output is the spam score; the spam
probability is sigmoid(score).
Rules the grader checks
Valuesupports+ - * /,-x,x ** n(n a plain number),exp,log,tanh,reluandsigmoid, with a plain number on either side (2 - xandx - 2both work).backward()sets the output's.gradto 1 and fills.gradfor every value in the graph. A value used in several places gets the sum of the gradients. Graphs can be a few hundred operations deep.reluhas gradient 0 at exactly 0.tanh(1000)is 1 andsigmoid(-1000)is 0, without errors.sigmoidis one operation with its own local slope.MLP(n_inputs, layer_sizes, activation="tanh"): hidden layers use the activation ("tanh"or"relu"), the last layer is linear, one output gives oneValue.parameters()returns each parameter once.Neuron(4)has 5 parameters;MLP(4, [8, 1])has 49.save_weightsandload_weightsfollow the format above exactly. The grader reads yourmlp_weights.jsonwith its own code.
The exact interface is in the "Interface" section at the end of this brief.
The core pieces you must be able to explain
You may use AI for anything in this quest. The defense then checks that you understand your own submission. Be ready to:
- trace
backward()by hand on a graph of three or four operations, including a reused value; - explain why gradients add up (
+=) and why the nodes are visited in reverse order; - give the local slope of any operation in your engine, for example
sigmoidortanh; - count the parameters of your network and say what the hidden layer does with the length clue;
- explain why a linear filter cannot make "rise, then fall", and why stacking linear layers without an activation does not help;
- defend your comparison: what was kept the same, which numbers changed, what accuracy hides;
- change your code: for example, add a new operation or a second hidden layer.
How you're graded
- Automated checks, on a public suite (you have it;
neugen testruns it) and a hidden one: - Gradients on expression graphs and Network checks: all must pass. The hidden suite adds deep graphs, values used many times, edge cases, relu networks and two hidden layers. - Your network, scored on the validation split (public) and on hidden test messages: loss at most 0.16 (no linear filter can get there), recall at least 0.68, precision at least 0.7. - Rise, then fall: your network must rate a 150-character message as more suspicious than a 50- or a 300-character one with the same other clues (no link, one prize word, 10% capitals), by at least 0.1 in probability. A linear filter always fails this. - Write-up review of
SUBMISSION.mdwith the rubric on this page. - Defense: a short conversation with the AI examiner about your code and results.
Suggested path (about 9 hours)
- Read the CS231n sections in the materials (1.5 hours). Trace
a * b + aon paper. - The length table (30 minutes).
engine.py, with finite-difference checks as you go (2.5 hours).nn.pyand the weights file (1.5 hours).- Training and choosing settings (1.5 hours).
- The comparison and a look inside (1 hour).
- Write-up,
neugen test, submit (30 minutes).
Tips
- Check every operation with finite differences the moment you write it. A bug in one local slope makes every network gradient wrong, and training then fails in confusing ways.
- Speed. Work out the clues once. A few thousand steps of 32 messages take a minute or two in pure Python for a 4-8-1 network; much bigger networks get slow.
- If training is stuck at the linear filter's loss (about 0.18), check that the inputs are standardized and that the gradients are reset each step, then try a learning rate 10 times larger or smaller. Print the validation loss every few hundred steps.
- Seeds. Starting weights and batches are random. Seed them so your numbers can be reproduced, and try more than one seed before you draw a conclusion.
- Accuracy hides the gain. The two models have almost the same accuracy, because 86% of messages are normal and most are easy. Look at loss and recall, and at the messages that changed.
- A bigger network is not automatically better. If the training loss keeps falling while the validation loss rises, the network is learning the training messages, not the pattern.
Interface
# engine.py
class Value:
data: float; grad: float # grad starts at 0.0
def __init__(self, data): ...
# a + b, a - b, a * b, a / b, -a, a ** n (n a plain int or float), and the same with a
# plain number on either side: 2 + a, 2 - a, 2 * a, 2 / a
def exp(self) -> "Value": ...
def log(self) -> "Value": ...
def tanh(self) -> "Value": ...
def relu(self) -> "Value": ... # gradient 0 at exactly 0
def sigmoid(self) -> "Value": ... # one operation; sigmoid(-1000) == 0.0, no overflow
def backward(self) -> None: ... # fills .grad of every node in the graph
# nn.py (activation is "tanh" or "relu"; the last layer is linear)
class Neuron: def __init__(self, n_inputs, activation="tanh"): ... # __call__(x) -> Value
class Layer: def __init__(self, n_inputs, n_outputs, activation="tanh"): ... # -> list
class MLP:
def __init__(self, n_inputs, layer_sizes, activation="tanh"): ...
input_mean: list[float]; input_std: list[float] # inputs become (x - mean) / std
def __call__(self, x): ... # one output -> one Value (the spam score)
def parameters(self) -> list: ... # each Value once
def save_weights(self, path): ... # the JSON format in nn.py and the brief
def load_weights(self, path): ... # into this network (same sizes and activation)
# each of Neuron, Layer, MLP has parameters()
# mlp_weights.json: your trained network, written by save_weights.
# A message is flagged as spam when its score is above 0 (probability above 0.5).
Read now
Read 'Quick intro', 'Single neuron as a linear classifier', 'Commonly used activation functions' (sigmoid, tanh and ReLU only), 'Layer-wise organization' and 'Example feed-forward computation'. Then find each piece (neuron, activation, hidden layer, output) in your 4-8-1 message classifier. Skip the biology and the later sections.
Explore further
Trace a small graph on paper before you code. Read 'Simple expressions and interpretation of the gradient', 'Compound expressions with chain rule', 'Intuitive understanding of backpropagation', 'Modularity: Sigmoid example' and 'Patterns in backward flow' (it explains why gradients add up where a value is used twice). Skip 'Gradients for vectorized operations'.
Three stages. You only move on when the one before passes.
Automated gates
| Metric | Gate | What it checks |
|---|---|---|
| Gradients on expression graphs | ≥ 1.0 | Share of expression graphs where the output and every input gradient are right. The hidden set uses values many times, a chain of 300 operations and edge cases (relu at 0, tanh and sigmoid of 1000 and -1000, powers, unused inputs). All must pass. |
| Network checks | ≥ 1.0 | Share of nn.py checks that pass: parameter counts, outputs, the forward pass under fixed weights loaded with load_weights, save_weights then load_weights, and gradients of a batch loss compared with finite differences. All must pass. |
| Your network: loss | ≤ 0.16 | Average cross-entropy of mlp_weights.json on validation (public) or the hidden test messages. The linear filter gets about 0.19 on validation and 0.175 on test, and no linear filter can reach 0.16. |
| Your network: spam caught (recall) | ≥ 0.68 | Share of spam messages with a score above 0. The linear filter catches about 66% on validation. |
| Your network: precision | ≥ 0.7 | Share of flagged messages that really are spam. |
| Rise, then fall | ≥ 0.1 | Spam probability of a 150-character message minus the larger of the probabilities at 50 and 300 characters, with the same other clues (no link, one prize word, 10% capitals). A linear filter always scores below 0 here. |
Gates run on the public set (which you have) and a hidden set with inputs you haven't seen.
Rubric review need ≥ 60%
"How my engine works" explains, in the student's own words and matching their code, what each Value stores from the forward pass, why nodes are processed in reverse topological order, and why gradients are added with += when a value is used more than once, with a small worked example. "Checking gradients" reports finite-difference checks the student ran themselves (at least one graph with a reused value and one network gradient), with the actual numbers.
"Spam by length" has the student's own length table (or plot) from the training split and states the pattern. "Network versus linear filter" compares the two on the same data, loss, validation messages and 0.5 rule, with loss, recall, precision and accuracy in a table produced by train.py (or a script like it); shows the length curve of both models; discusses a few messages they disagree on, including one the network gets wrong; says what accuracy hides; and keeps the conclusion within the evidence (for example, one seed or a gain of a few messages is not proof). Choices such as the hidden size, learning rate, steps and standardization are explained with what was tried.
The logs and "How I used and checked AI" show the student used AI to learn (for example asked it to trace a graph, or to explain why stacked linear layers stay linear) and checked what it produced: a local derivative verified with finite differences, AI-written code tested on a hand-worked example, a claim about the data checked against their own table, or a mistake found and fixed. Using AI to write code is fine; judge the checking.
Defense need ≥ 50%
A 4-question conversation with the AI examiner about your code. It isn't a trick: if you did it and understood it, it's the easy part. A weak defense on strong work gets you a new variant to redo.

