Learn
Language 🤖 AI-on 💻 On your computer ⏱ ~10h Difficulty 3/5 v1

Teach your message filter to read the words

Four hand-picked clues miss a third of unwanted messages. Teach the filter to read: split messages into tokens, build a vocabulary from the training messages, turn each message into word counts and learn one weight per word. This is where PyTorch arrives: check its gradients against your own engine, then train a model that classifies raw messages.

Join free to start → Read the brief and materials. Join AI from Scratch to start, submit and ask the tutor. Log in

The problem: clues are not enough

Your filter from the last quests looks at four clues: a link, prize words, capital letters and length. Here are three unwanted messages from the validation file that it lets through:

Please CALL 08712402902 immediately as there is an urgent message waiting for you.

You have 1 new voicemail. Please call 08719181513.

Dear U've been invited to XCHAT. This is our final attempt to contact u! Txt CHAT to 86688

No link, almost no prize words, short. But you can tell they are unwanted at once, because you read the words: "call", "urgent", "voicemail", "txt", "chat". And you know these are normal:

R we still meeting 4 dinner tonight?

K give me a sec, breaking a <#> at cstore

On the validation messages, the four-clue filter catches only about 2 in 3 unwanted messages. In this quest your filter learns from the words themselves. It decides which words matter, and how much, from the training messages. No more hand-picked clues.

This is also where you meet PyTorch, the library most AI models are built with. You will see why you need it: reading words means thousands of weights, and plain Python gets slow.

What you build

"Claim your FREE prize!"
  -> tokens:        ["claim", "your", "free", "prize", "!"]
  -> IDs:           [118, 21, 47, 205, 9]
  -> counts:        a list with one number per word in the vocabulary (mostly zeros)
  -> score:         counts . weights + bias      (one weight per word)
  -> decision:      sigmoid(score) = 0.97  ->  unwanted
  1. tokenizer.py: a tokenizer that splits a message into tokens, and a vocabulary that gives every known token an ID, saved to vocab.json.
  2. check_autograd.py: the same small classifier computed twice, with your engine from the last quest and with PyTorch, to check that they give the same gradients.
  3. classifier.py: bag-of-words counts and a PyTorch model, trained and saved to classifier.pt, with a predict() that takes raw messages.
  4. An honest comparison with the four-clue filter, and a look at the mistakes.

The next quest (autocomplete) reuses your tokenizer, so keep the names in tokenizer.py.

Set up PyTorch

pip install -r requirements.txt      # installs torch==2.14.1 (Windows, Apple Silicon Macs, Linux)

PyTorch 2.14.1 needs Python 3.10 or newer.

  • Intel Mac: the last PyTorch for Intel Macs is 2.2.2. Run pip install torch==2.2.2 instead (it needs Python 3.12 or older). Everything in this quest works with it. If you see a long warning about NumPy when you import torch, it is harmless (or run pip install "numpy<2").
  • Linux without a graphics card: pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu saves a download of several gigabytes.

Check it: python -c "import torch; print(torch.__version__)". You do not need a GPU.

Part 1: from text to tokens to IDs

Tokens. A computer sees a message as a string of characters. A tokenizer cuts it into pieces called tokens: here, words and punctuation marks.

"R we still meeting 4 dinner tonight?"
  -> ["r", "we", "still", "meeting", "4", "dinner", "tonight", "?"]

You decide the rules (your normalization policy). Some rules are fixed, because the grader and the next quest rely on them:

  • Everything is lowercase: "FREE", "Free" and "free" are the same token.
  • The placeholder <#> stays one token. (In this dataset, some numbers were replaced by <#>.)
  • Punctuation never sticks to a word, and it is kept: "hello," gives "hello" and ",". ("!" is a useful clue for spam, so don't throw it away.)
  • A word is a run of letters and digits: "2nite" is one token.
  • Spaces only separate tokens. An empty message gives [].

The rest is your choice: apostrophes ("don't"), runs like "!!!", curly quotes, web addresses. Look at real messages in data/sms_train.tsv and decide. Write your policy at the top of tokenizer.py. A regular expression is the usual tool (see the reading).

Vocabulary and IDs. A vocabulary is the list of tokens your model knows. Each token gets a number, its ID. Build it from the training messages only, keeping tokens that appear at least min_count times:

training tokens:  can we meet ?   we can .   see you ?        (min_count = 2)
vocabulary:       0 <unk>   1 <s>   2 </s>   3 can   4 we   5 ?
  • "<unk>" (ID 0) stands for every unknown token: a word the vocabulary does not have. A word that appears only in the validation or test messages must stay unknown. If you let test words into the vocabulary, your filter has seen the test before it is tested, and its scores lie.
  • "<s>" and "</s>" (IDs 1 and 2) mark the start and end of a message. You don't need them here; the autocomplete quest does. Reserve them now so IDs never change.
  • Your other tokens start at ID 3, in an order that is the same every time you build from the same data (for example, most frequent first, ties in alphabetical order).
  • save() writes the vocabulary to JSON, and load() reads it back with exactly the same IDs. If IDs moved, your saved weights would belong to the wrong words.

decode() gives back tokens, not the original message: decode(encode(tokenize("Hi Bob!"))) is ["hi", "<unk>", "!"] if "bob" is unknown. The case and the spaces are gone too. That's fine.

Part 2: bag of words, and why plain Python gets slow

IDs are names, not amounts. ID 205 is not "more" than ID 21. So the model doesn't get the IDs. It gets counts: a list with one place per vocabulary token, holding how often that token appears in the message. This is called a bag of words.

vocabulary size 6:   0 <unk>  1 <s>  2 </s>  3 can  4 we  5 ?
"can we? we can!"  -> tokens  can we ? we can !    -> IDs  3 4 5 4 3 0
bag of words       -> [1, 0, 0, 2, 2, 1]           ("!" is unknown: counted at ID 0)

The score is the same formula as your earlier filter, just with many more inputs: score = count_0 * w_0 + count_1 * w_1 + ... + b, one weight per vocabulary token. A positive weight means "this word suggests spam". It is logistic regression again, with thousands of clues instead of four.

Notice what is lost: "Alice owes Bob" and "Bob owes Alice" have the same counts. The bag of words forgets the order. (The tokenizer keeps it; only the counts lose it.)

The speed problem. With about 3,500 tokens in the vocabulary and 3,900 training messages, one pass over the data is more than 13 million multiplications. In your engine, every multiplication creates a Value object. Try it on a small sample (say 50 messages) and time one training step with your engine. Most of those counts are zero, and Python handles one number at a time.

Tensors. A tensor is a block of numbers with a shape. A list of 6 counts is a tensor of shape (6,). All training messages together are a tensor of shape (3900, vocab_size): one row per message. PyTorch does arithmetic on whole tensors at once, in fast compiled code:

X = torch.tensor([[1., 0., 2.],     # shape (2, 3): 2 messages, 3 tokens
                  [0., 1., 1.]])
w = torch.tensor([0.5, -1.0, 0.25]) # shape (3,): one weight per token
X @ w + 0.1                         # shape (2,): one score per message -> tensor([1.1000, -0.6500])

X @ w is a matrix-vector product: row 1 gives 10.5 + 0(-1.0) + 2*0.25 = 1.0, plus the bias 0.1 gives 1.1. One line replaces two nested loops. Rewrite your classifier's scores and loss with tensors, and time it against the scalar version on the same sample. Write both timings down.

Part 3: let PyTorch compute the gradients

In the last quest you built backward(): every Value remembers how it was made, and the chain rule runs backwards through the graph. PyTorch does the same for tensors. It is called autograd:

w = torch.tensor([0.5, -1.0, 0.25], requires_grad=True)   # "track how w is used"
loss = ((X @ w) ** 2).mean()                              # PyTorch records the graph as it computes
loss.backward()                                           # the chain rule, backwards
w.grad                                                    # d loss / d w, one number per weight

Before you trust it, check it. In check_autograd.py, compute the mean loss of a tiny logistic regression and its gradients twice: once with your engine (engine.py, provided from the last quest), once with tensors and loss.backward(). Run python check_autograd.py: the two lines must match. You can also check one gradient by hand: for one message, the gradient of the loss with respect to the bias is p - y (the predicted probability minus the label).

Part 4: train it the PyTorch way

Now write MessageClassifier, a small nn.Module with one nn.Linear layer. A module holds the parameters (the weights and the bias) and says how to compute scores from counts. An optimizer from torch.optim (for example SGD or Adam) does the update step you wrote by hand in quest 3. One training step is always the same four lines:

optimizer.zero_grad()                 # forget the gradients of the last step
loss = loss_fn(model(X), y)           # forward: scores, then the loss
loss.backward()                       # backward: gradients for every parameter
optimizer.step()                      # update: move each parameter against its gradient

Train on the training messages, choose your settings with the validation messages, then save:

vocab.save("vocab.json")
torch.save(model.state_dict(), "classifier.pt")    # the weights, by name

predict(texts, vocab, model) takes raw messages and returns 0 or 1 for each one, with a threshold you choose. load_filter(folder) (given) loads your two files. Then look inside: which tokens have the largest positive and negative weights? Do they make sense?

Part 5: compare honestly

Train the four-clue filter on the same split (your PyTorch code works for it too: 4 inputs instead of thousands; features.py is included). Report accuracy, spam recall and spam precision for both on the validation messages. (Recall: the share of unwanted messages you catch. Precision: the share of flagged messages that really are unwanted.)

Then read the mistakes. Which unwanted messages still get through, and which normal messages get flagged? Print their tokens. Is it unknown words, a word that is innocent in one message and suspicious in another, or the lost word order? Better scores are likely here, but not guaranteed on every message. Say what you see.

What to submit

  • tokenizer.py, classifier.py, check_autograd.py, and your training code.
  • vocab.json and classifier.pt (both small: well under 1 MB).
  • SUBMISSION.md with these sections (the starter has a template): 1. Tokenizer and vocabulary: your policy and why; tricky messages; how many validation tokens are unknown. 2. From plain Python to PyTorch: your timings; how you checked the gradients. 3. Results and an honest comparison: a table for both filters on the validation messages. 4. Mistakes and what the counts miss: real examples, and the word-order problem. 5. How I used AI and checked it.
  • Your AI transcripts in .neugen/logs/ (neugen log <file>).

How you're graded

  1. Hidden tests (all run on your submitted files; neugen test runs the public ones): - Tokenizer (all must pass): lowercase, <#>, punctuation, empty and unusual messages, word order. - Vocabulary (all must pass): min_count, ID 0 for unknown tokens, fixed special IDs, the same IDs after save and load, and your vocab.json contains training words only. - Model (all must pass): bag-of-words counts, one linear layer, your artifacts load, and predict() follows your model's scores on any message. - Autograd (all must pass): both functions give the right loss and gradients on small examples. - Quality on messages you have never seen: spam recall at least 0.80, spam precision at least 0.90, accuracy at least 0.97. A careful filter scores around 0.9 or more on all three. The hidden messages are different, so don't tune until validation is just above the line.
  2. Rubric review by an AI grader: your comparison and mistakes, your move to PyTorch, and how you used and checked AI.
  3. Defense: a short conversation about your code.

What you must be able to explain and change

This is an AI-on quest: use AI for anything, including writing code. But in the defense you will be asked about these, in your own code:

  • your tokenizer's rules, and what it does with a message you are shown;
  • how the vocabulary is built, why only from training messages, and what happens to an unknown word;
  • bag_of_words on a tiny example, by hand;
  • the shapes in your model: counts (batch, vocab_size) -> scores (batch,);
  • what loss.backward() and optimizer.step() do, and how you checked autograd against your engine;
  • a small change, for example a new tokenizer rule, a different min_count, or a different threshold, and what it would do to your results.

Suggested path (about 10 hours)

  1. Read the two materials (2 hours).
  2. Tokenizer and vocabulary, tried on real messages (2 hours).
  3. Bag of words; time the engine against tensors on a small sample (1.5 hours).
  4. check_autograd.py (1 hour).
  5. Train with nn.Module and an optimizer; save; neugen test (1.5 hours).
  6. Compare, read mistakes, write SUBMISSION.md (2 hours).

Tips

  • Look at the data first. Print 20 messages and their tokens. You will find <#>, "u", "2", "...", "£" and web addresses. Your rules come from what you see.
  • Count unknown tokens. If a large share of validation tokens are unknown, try a different min_count and see what happens to your scores.
  • Use float tensors. bag_of_words should return a float tensor (torch.float32), because the linear layer multiplies it by float weights.
  • Shapes cause most bugs. Print .shape after every step until it stops surprising you.
  • Never fit on validation or test messages: not the vocabulary, not the weights.

Interface

# tokenizer.py  (the next quest reuses it: keep the names)
def tokenize(text: str) -> list[str]
class Vocab:
    UNK, BOS, EOS = "<unk>", "<s>", "</s>"          # IDs 0, 1, 2; your tokens start at 3
    @classmethod
    def build(cls, token_lists: list[list[str]], min_count: int = 1) -> "Vocab"
    def encode(self, tokens: list[str]) -> list[int]  # unknown token -> 0
    def decode(self, ids: list[int]) -> list[str]
    def __len__(self) -> int
    def save(self, path) -> None                      # JSON
    @classmethod
    def load(cls, path) -> "Vocab"

# classifier.py
def bag_of_words(ids: list[int], vocab_size: int) -> torch.Tensor   # float, shape (vocab_size,)
class MessageClassifier(nn.Module):
    def __init__(self, vocab_size: int)                # one nn.Linear(vocab_size, 1)
    def forward(self, counts) -> torch.Tensor          # (batch, vocab_size) -> (batch,) scores
def load_filter(folder=".") -> tuple[Vocab, MessageClassifier]      # vocab.json + classifier.pt
def predict(texts: list[str], vocab, model) -> list[int]           # 1 = spam, 0 = normal

# check_autograd.py  (mean binary cross-entropy of sigmoid(X @ w + b))
def engine_loss_and_grads(w, b, X, y) -> tuple[float, list[float], float]   # with engine.Value
def torch_loss_and_grads(w, b, X, y) -> tuple[float, list[float], float]    # with torch.autograd

# submit: vocab.json and classifier.pt (torch.save(model.state_dict(), ...)), well under 1 MB