Teach your message filter to read the words
Four hand-picked clues miss a third of unwanted messages. Teach the filter to read: split messages into tokens, build a vocabulary from the training messages, turn each message into word counts and learn one weight per word. This is where PyTorch arrives: check its gradients against your own engine, then train a model that classifies raw messages.
The problem: clues are not enough
Your filter from the last quests looks at four clues: a link, prize words, capital letters and length. Here are three unwanted messages from the validation file that it lets through:
Please CALL 08712402902 immediately as there is an urgent message waiting for you.
You have 1 new voicemail. Please call 08719181513.
Dear U've been invited to XCHAT. This is our final attempt to contact u! Txt CHAT to 86688
No link, almost no prize words, short. But you can tell they are unwanted at once, because you read the words: "call", "urgent", "voicemail", "txt", "chat". And you know these are normal:
R we still meeting 4 dinner tonight?
K give me a sec, breaking a <#> at cstore
On the validation messages, the four-clue filter catches only about 2 in 3 unwanted messages. In this quest your filter learns from the words themselves. It decides which words matter, and how much, from the training messages. No more hand-picked clues.
This is also where you meet PyTorch, the library most AI models are built with. You will see why you need it: reading words means thousands of weights, and plain Python gets slow.
What you build
"Claim your FREE prize!"
-> tokens: ["claim", "your", "free", "prize", "!"]
-> IDs: [118, 21, 47, 205, 9]
-> counts: a list with one number per word in the vocabulary (mostly zeros)
-> score: counts . weights + bias (one weight per word)
-> decision: sigmoid(score) = 0.97 -> unwanted
tokenizer.py: a tokenizer that splits a message into tokens, and a vocabulary that gives every known token an ID, saved tovocab.json.check_autograd.py: the same small classifier computed twice, with your engine from the last quest and with PyTorch, to check that they give the same gradients.classifier.py: bag-of-words counts and a PyTorch model, trained and saved toclassifier.pt, with apredict()that takes raw messages.- An honest comparison with the four-clue filter, and a look at the mistakes.
The next quest (autocomplete) reuses your tokenizer, so keep the names in tokenizer.py.
Set up PyTorch
pip install -r requirements.txt # installs torch==2.14.1 (Windows, Apple Silicon Macs, Linux)
PyTorch 2.14.1 needs Python 3.10 or newer.
- Intel Mac: the last PyTorch for Intel Macs is 2.2.2. Run
pip install torch==2.2.2instead (it needs Python 3.12 or older). Everything in this quest works with it. If you see a long warning about NumPy when you import torch, it is harmless (or runpip install "numpy<2"). - Linux without a graphics card:
pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpusaves a download of several gigabytes.
Check it: python -c "import torch; print(torch.__version__)". You do not need a GPU.
Part 1: from text to tokens to IDs
Tokens. A computer sees a message as a string of characters. A tokenizer cuts it into pieces called tokens: here, words and punctuation marks.
"R we still meeting 4 dinner tonight?"
-> ["r", "we", "still", "meeting", "4", "dinner", "tonight", "?"]
You decide the rules (your normalization policy). Some rules are fixed, because the grader and the next quest rely on them:
- Everything is lowercase: "FREE", "Free" and "free" are the same token.
- The placeholder
<#>stays one token. (In this dataset, some numbers were replaced by<#>.) - Punctuation never sticks to a word, and it is kept:
"hello,"gives"hello"and",". ("!"is a useful clue for spam, so don't throw it away.) - A word is a run of letters and digits:
"2nite"is one token. - Spaces only separate tokens. An empty message gives
[].
The rest is your choice: apostrophes ("don't"), runs like "!!!", curly quotes, web
addresses. Look at real messages in data/sms_train.tsv and decide. Write your policy at the top
of tokenizer.py. A regular expression is the usual tool (see the reading).
Vocabulary and IDs. A vocabulary is the list of tokens your model knows. Each token gets a
number, its ID. Build it from the training messages only, keeping tokens that appear at
least min_count times:
training tokens: can we meet ? we can . see you ? (min_count = 2)
vocabulary: 0 <unk> 1 <s> 2 </s> 3 can 4 we 5 ?
"<unk>"(ID 0) stands for every unknown token: a word the vocabulary does not have. A word that appears only in the validation or test messages must stay unknown. If you let test words into the vocabulary, your filter has seen the test before it is tested, and its scores lie."<s>"and"</s>"(IDs 1 and 2) mark the start and end of a message. You don't need them here; the autocomplete quest does. Reserve them now so IDs never change.- Your other tokens start at ID 3, in an order that is the same every time you build from the same data (for example, most frequent first, ties in alphabetical order).
save()writes the vocabulary to JSON, andload()reads it back with exactly the same IDs. If IDs moved, your saved weights would belong to the wrong words.
decode() gives back tokens, not the original message: decode(encode(tokenize("Hi Bob!")))
is ["hi", "<unk>", "!"] if "bob" is unknown. The case and the spaces are gone too. That's fine.
Part 2: bag of words, and why plain Python gets slow
IDs are names, not amounts. ID 205 is not "more" than ID 21. So the model doesn't get the IDs. It gets counts: a list with one place per vocabulary token, holding how often that token appears in the message. This is called a bag of words.
vocabulary size 6: 0 <unk> 1 <s> 2 </s> 3 can 4 we 5 ?
"can we? we can!" -> tokens can we ? we can ! -> IDs 3 4 5 4 3 0
bag of words -> [1, 0, 0, 2, 2, 1] ("!" is unknown: counted at ID 0)
The score is the same formula as your earlier filter, just with many more inputs:
score = count_0 * w_0 + count_1 * w_1 + ... + b, one weight per vocabulary token. A positive
weight means "this word suggests spam". It is logistic regression again, with thousands of
clues instead of four.
Notice what is lost: "Alice owes Bob" and "Bob owes Alice" have the same counts. The bag of words forgets the order. (The tokenizer keeps it; only the counts lose it.)
The speed problem. With about 3,500 tokens in the vocabulary and 3,900 training messages,
one pass over the data is more than 13 million multiplications. In your engine, every
multiplication creates a Value object. Try it on a small sample (say 50 messages) and time one
training step with your engine. Most of those counts are zero, and Python handles one number at
a time.
Tensors. A tensor is a block of numbers with a shape. A list of 6 counts is a tensor
of shape (6,). All training messages together are a tensor of shape (3900, vocab_size): one
row per message. PyTorch does arithmetic on whole tensors at once, in fast compiled code:
X = torch.tensor([[1., 0., 2.], # shape (2, 3): 2 messages, 3 tokens
[0., 1., 1.]])
w = torch.tensor([0.5, -1.0, 0.25]) # shape (3,): one weight per token
X @ w + 0.1 # shape (2,): one score per message -> tensor([1.1000, -0.6500])
X @ w is a matrix-vector product: row 1 gives 10.5 + 0(-1.0) + 2*0.25 = 1.0, plus the
bias 0.1 gives 1.1. One line replaces two nested loops. Rewrite your classifier's scores and
loss with tensors, and time it against the scalar version on the same sample. Write both
timings down.
Part 3: let PyTorch compute the gradients
In the last quest you built backward(): every Value remembers how it was made, and the chain
rule runs backwards through the graph. PyTorch does the same for tensors. It is called
autograd:
w = torch.tensor([0.5, -1.0, 0.25], requires_grad=True) # "track how w is used"
loss = ((X @ w) ** 2).mean() # PyTorch records the graph as it computes
loss.backward() # the chain rule, backwards
w.grad # d loss / d w, one number per weight
Before you trust it, check it. In check_autograd.py, compute the mean loss of a tiny
logistic regression and its gradients twice: once with your engine (engine.py, provided from
the last quest), once with tensors and loss.backward(). Run python check_autograd.py: the two
lines must match. You can also check one gradient by hand: for one message, the gradient of the
loss with respect to the bias is p - y (the predicted probability minus the label).
Part 4: train it the PyTorch way
Now write MessageClassifier, a small nn.Module with one nn.Linear layer. A module
holds the parameters (the weights and the bias) and says how to compute scores from counts.
An optimizer from torch.optim (for example SGD or Adam) does the update step you wrote
by hand in quest 3. One training step is always the same four lines:
optimizer.zero_grad() # forget the gradients of the last step
loss = loss_fn(model(X), y) # forward: scores, then the loss
loss.backward() # backward: gradients for every parameter
optimizer.step() # update: move each parameter against its gradient
Train on the training messages, choose your settings with the validation messages, then save:
vocab.save("vocab.json")
torch.save(model.state_dict(), "classifier.pt") # the weights, by name
predict(texts, vocab, model) takes raw messages and returns 0 or 1 for each one, with a
threshold you choose. load_filter(folder) (given) loads your two files. Then look inside:
which tokens have the largest positive and negative weights? Do they make sense?
Part 5: compare honestly
Train the four-clue filter on the same split (your PyTorch code works for it too: 4 inputs
instead of thousands; features.py is included). Report accuracy, spam recall and spam
precision for both on the validation messages. (Recall: the share of unwanted messages you
catch. Precision: the share of flagged messages that really are unwanted.)
Then read the mistakes. Which unwanted messages still get through, and which normal messages get flagged? Print their tokens. Is it unknown words, a word that is innocent in one message and suspicious in another, or the lost word order? Better scores are likely here, but not guaranteed on every message. Say what you see.
What to submit
tokenizer.py,classifier.py,check_autograd.py, and your training code.vocab.jsonandclassifier.pt(both small: well under 1 MB).SUBMISSION.mdwith these sections (the starter has a template): 1. Tokenizer and vocabulary: your policy and why; tricky messages; how many validation tokens are unknown. 2. From plain Python to PyTorch: your timings; how you checked the gradients. 3. Results and an honest comparison: a table for both filters on the validation messages. 4. Mistakes and what the counts miss: real examples, and the word-order problem. 5. How I used AI and checked it.- Your AI transcripts in
.neugen/logs/(neugen log <file>).
How you're graded
- Hidden tests (all run on your submitted files;
neugen testruns the public ones): - Tokenizer (all must pass): lowercase,<#>, punctuation, empty and unusual messages, word order. - Vocabulary (all must pass):min_count, ID 0 for unknown tokens, fixed special IDs, the same IDs after save and load, and yourvocab.jsoncontains training words only. - Model (all must pass): bag-of-words counts, one linear layer, your artifacts load, andpredict()follows your model's scores on any message. - Autograd (all must pass): both functions give the right loss and gradients on small examples. - Quality on messages you have never seen: spam recall at least 0.80, spam precision at least 0.90, accuracy at least 0.97. A careful filter scores around 0.9 or more on all three. The hidden messages are different, so don't tune until validation is just above the line. - Rubric review by an AI grader: your comparison and mistakes, your move to PyTorch, and how you used and checked AI.
- Defense: a short conversation about your code.
What you must be able to explain and change
This is an AI-on quest: use AI for anything, including writing code. But in the defense you will be asked about these, in your own code:
- your tokenizer's rules, and what it does with a message you are shown;
- how the vocabulary is built, why only from training messages, and what happens to an unknown word;
bag_of_wordson a tiny example, by hand;- the shapes in your model: counts
(batch, vocab_size)-> scores(batch,); - what
loss.backward()andoptimizer.step()do, and how you checked autograd against your engine; - a small change, for example a new tokenizer rule, a different
min_count, or a different threshold, and what it would do to your results.
Suggested path (about 10 hours)
- Read the two materials (2 hours).
- Tokenizer and vocabulary, tried on real messages (2 hours).
- Bag of words; time the engine against tensors on a small sample (1.5 hours).
check_autograd.py(1 hour).- Train with
nn.Moduleand an optimizer; save;neugen test(1.5 hours). - Compare, read mistakes, write
SUBMISSION.md(2 hours).
Tips
- Look at the data first. Print 20 messages and their tokens. You will find
<#>, "u", "2", "...", "£" and web addresses. Your rules come from what you see. - Count unknown tokens. If a large share of validation tokens are unknown, try a different
min_countand see what happens to your scores. - Use float tensors.
bag_of_wordsshould return a float tensor (torch.float32), because the linear layer multiplies it by float weights. - Shapes cause most bugs. Print
.shapeafter every step until it stops surprising you. - Never fit on validation or test messages: not the vocabulary, not the weights.
Interface
# tokenizer.py (the next quest reuses it: keep the names)
def tokenize(text: str) -> list[str]
class Vocab:
UNK, BOS, EOS = "<unk>", "<s>", "</s>" # IDs 0, 1, 2; your tokens start at 3
@classmethod
def build(cls, token_lists: list[list[str]], min_count: int = 1) -> "Vocab"
def encode(self, tokens: list[str]) -> list[int] # unknown token -> 0
def decode(self, ids: list[int]) -> list[str]
def __len__(self) -> int
def save(self, path) -> None # JSON
@classmethod
def load(cls, path) -> "Vocab"
# classifier.py
def bag_of_words(ids: list[int], vocab_size: int) -> torch.Tensor # float, shape (vocab_size,)
class MessageClassifier(nn.Module):
def __init__(self, vocab_size: int) # one nn.Linear(vocab_size, 1)
def forward(self, counts) -> torch.Tensor # (batch, vocab_size) -> (batch,) scores
def load_filter(folder=".") -> tuple[Vocab, MessageClassifier] # vocab.json + classifier.pt
def predict(texts: list[str], vocab, model) -> list[int] # 1 = spam, 0 = normal
# check_autograd.py (mean binary cross-entropy of sigmoid(X @ w + b))
def engine_loss_and_grads(w, b, X, y) -> tuple[float, list[float], float] # with engine.Value
def torch_loss_and_grads(w, b, X, y) -> tuple[float, list[float], float] # with torch.autograd
# submit: vocab.json and classifier.pt (torch.save(model.state_dict(), ...)), well under 1 MB
Read now
Read 2.1 'Words' (pages 2-5: types, tokens, what counts as a word) and 2.6 'Regular Expressions' up to and including 2.6.5 'A Simple Example' (pages 16-21), then 2.8 'Rule-based Tokenization' (pages 26-27). Skip Unicode, BPE and edit distance. You need just enough regular expressions to write your tokenizer.
Read only four pages: 'Tensors', 'Build Model', 'Automatic Differentiation' and 'Optimization Loop'. Skip the datasets and images parts. While you read, think of your message filter: counts in, one score out.
Three stages. You only move on when the one before passes.
Automated gates
| Metric | Gate | What it checks |
|---|---|---|
| Tokenizer | ≥ 1.0 | Share of tokenizer checks that pass: lowercase, <#> as one token, punctuation split off and kept, words of letters and digits, empty and unusual messages, word order. All must pass. |
| Vocabulary | ≥ 1.0 | Share of vocabulary checks that pass: min_count, ID 0 for unknown tokens, IDs 0-2 for <unk> <s> </s>, the same IDs every time and after save/load, and your vocab.json has training words only. All must pass. |
| Bag of words and model | ≥ 1.0 | Share of checks that pass: bag_of_words counts, one linear layer giving (batch,) scores, your vocab.json and classifier.pt load, predict() works on any message and follows your model's scores. All must pass. |
| Engine and autograd agree | ≥ 1.0 | Share of small examples where both engine_loss_and_grads and torch_loss_and_grads give the right loss and gradients. All must pass. |
| Spam recall | ≥ 0.8 | Share of unwanted messages that predict() flags, on messages your model never saw. The four-clue filter catches about 0.66. |
| Spam precision | ≥ 0.9 | Share of flagged messages that really are unwanted. |
| Accuracy | ≥ 0.97 | Share of all messages classified correctly. Always answering 'normal' already scores about 0.86. |
Gates run on the public set (which you have) and a hidden set with inputs you haven't seen.
Rubric review need ≥ 60%
A table of accuracy, spam recall and spam precision for the word filter and the four-clue filter, trained and measured on the same split (validation, not test). Real misclassified messages with their tokens, and a specific reason for each (an unknown word, an innocent word in a spam message, the lost order). The write-up explains with an example that counts lose word order, and does not claim more than the numbers show.
The student timed the scalar version against tensors on the same sample and reports both numbers; explains why tensors are faster; describes how they checked torch.autograd against their engine (which examples, what matched) and states the shapes in their model. Their tokenizer policy is written down with reasons tied to real messages.
The logs show the student used AI to understand and to build, then checked it: asked for explanations of tokens, tensors or autograd; verified a gradient or a bag of words by hand; ran code and read the output; caught and fixed at least one wrong answer or bug from the AI (for example a vocabulary built from all the data, a wrong shape, a tokenizer rule that broke on a real message). Using AI is expected and never penalised. Judge the logs, not the claims; missing logs cap this criterion at 1.
Defense need ≥ 50%
A 4-question conversation with the AI examiner about your code. It isn't a trick: if you did it and understood it, it's the easy part. A weak defense on strong work gets you a new variant to redo.

