Build a message filter
Unwanted messages are annoying and sometimes dangerous. Build a filter that gives each message a score from a few clues (a link, prize words, capital letters, length), choose its weights by hand, measure how well it works and read the messages it gets wrong. Your first model: a function whose behaviour is set by numbers you choose.
Why this quest
Your phone receives three messages:
| Message | |
|---|---|
| A | Congratulations! You have won a £500 prize. To claim, call 09061701461 before midnight. Ref: KL341 |
| B | Your order 4471 has shipped. Track it at www.parcelhub.example.com/track. Expected delivery: Thursday. |
| C | Hi Ana, can we move tomorrow's meeting to 3pm? Same room. Thanks, Kenji |
You know at once that A is unwanted (spam) and that B and C are normal. But how do you know? Perhaps the prize, the money sign, the urgent "before midnight". B has a link, like a lot of spam, but it is still a normal message. A computer needs numbers and a rule.
In this quest you build that rule: a small message filter. It looks at a few clues about a message, gives the message a score, and flags it as spam when the score is high enough. You choose the filter's numbers by hand, measure how well it works on real messages, and read the messages it gets wrong. This is your first model: a function whose behaviour is set by numbers you choose. In the next quest, the computer chooses those numbers for you.
1. Clues: numbers that describe a message
The starter kit's features.py turns a message into four numbers, called clues (or
features):
| Clue | Meaning | A | B | C |
|---|---|---|---|---|
has_link |
1 if it contains a web address, else 0 | 0 | 1 | 0 |
prize_words |
how many prize or money words (free, win, prize, cash, claim, £...) | 5 | 0 | 0 |
caps_fraction |
share of its letters that are capitals (0 to 1) | 0.0984 | 0.0506 | 0.0943 |
length |
its length in characters, divided by 100 | 0.98 | 1.02 | 0.71 |
from features import extract
extract("Congratulations! You have won a £500 prize. ...") # -> [0.0, 5.0, 0.0984, 0.98]
You don't need to change features.py. The grader uses its own copy.
2. Parameters: numbers that control the filter
Here is a question to keep in mind for the whole quest: which numbers describe the message, and which numbers control the filter?
The clues describe the message. You can't change them. The filter has its own numbers, which you choose: one weight per clue and one bias. These are its parameters.
The filter's score for a message is:
score = bias + weight1 * has_link + weight2 * prize_words + weight3 * caps_fraction + weight4 * length
A worked example. Take weights [2, 1.5, 3, 0.5] and bias -2:
A: -2 + 2*0 + 1.5*5 + 3*0.0984 + 0.5*0.98 = -2 + 0 + 7.5 + 0.2952 + 0.49 = 6.2852
B: -2 + 2*1 + 1.5*0 + 3*0.0506 + 0.5*1.02 = -2 + 2 + 0 + 0.1518 + 0.51 = 0.6618
C: -2 + 2*0 + 1.5*0 + 3*0.0943 + 0.5*0.71 = -2 + 0 + 0 + 0.2829 + 0.355 = -1.3621
A big positive weight means "this clue makes a message more suspicious". A weight of 0 means "ignore this clue". A negative weight means "this clue makes a message look more normal". The bias is the starting point before any clue is counted: a negative bias means "a message is normal until the clues say otherwise".
To decide, compare the score with a threshold. If the score is greater than the threshold, the filter predicts 1 (spam); otherwise it predicts 0 (normal). With threshold 0, this filter flags A (6.29) and leaves C (-1.36). But it also flags B (0.66), the order notice. Which number would you change to fix B, and what might that break? That is the work of this quest.
Note: a score is not a probability. A score of 6.29 does not mean "629% sure". It only says "more suspicious than a message with a lower score". (The next quest turns scores into probabilities.)
3. Measuring a filter
Every message in the data has a label: 1 for spam, 0 for normal. People wrote these labels, not the filter. Compare each prediction with its label, and count four kinds of result:
| the filter says spam (1) | the filter says normal (0) | |
|---|---|---|
| really spam | true positive (tp) | false negative (fn): missed |
| really normal | false positive (fp): a false alarm | true negative (tn) |
From these counts:
- accuracy = (tp + tn) / all messages: the share it gets right.
- precision = tp / (tp + fp): of the messages it flags, the share that really are spam.
- recall = tp / (tp + fn): of the spam messages, the share it catches.
A small example: 10 messages, 2 of them spam. The filter flags 3 messages: 1 spam and 2 normal. Then tp = 1, fp = 2, fn = 1, tn = 6. Accuracy = 7/10 = 0.7, precision = 1/3 ≈ 0.33, recall = 1/2 = 0.5.
Why three numbers? About 86% of the messages in this data are normal. A "filter" that never flags anything has 86% accuracy, and catches no spam at all: recall 0. Always look at recall and precision, not only accuracy.
When nothing is flagged, precision would divide by zero. In this quest, precision is 0.0 when nothing is flagged, and recall is 0.0 when there is no spam in the list.
4. What to build
Two files, in your starter kit:
filter.py, with three functions (the docstrings explain each one):
def score(clues, weights, bias) -> float # bias + the sum of weight * clue
def predict(clues, weights, bias, threshold=0.0) -> int # 1 if score > threshold, else 0
def evaluate(messages, weights, bias, threshold=0.0) -> dict
messages is a list of (text, label) pairs, as features.load_messages("data/sms_val.tsv")
returns them. evaluate gets each message's clues with features.extract(text) and returns a
dict with at least tp, fp, fn, tn, accuracy, precision and recall. Keep these
names exactly: the next quests use them.
my_filter.json: the filter you choose by hand.
{"weights": [2, 1.5, 3, 0.5], "bias": -2, "threshold": 0}
Then choose your parameters. Read some spam and normal messages in data/sms_train.tsv.
Think about each clue: how often does it appear in spam, and in normal messages? Pick weights,
measure, read the mistakes, change one number, measure again. python try_filter.py prints the
numbers and some mistakes on the validation messages. Keep notes of what you tried: you will need
them for your write-up.
You don't need a perfect filter. The goal is to understand what each number does. A thoughtful filter catches well over half of the spam on these four clues; no choice of weights catches all of it. Find out why.
5. The data
data/ holds real text messages from the public SMS Spam Collection (collected in 2011,
mostly from the UK and Singapore; see data/SOURCE.md):
sms_train.tsv(3,900 messages): read these and use them to choose your weights.sms_val.tsv(833 messages): check your filter on these (try_filter.pyuses them).- A hidden test set (841 messages) that you never see. The grader measures your filter on it.
Each line is label<TAB>text. Messages use 2011 text-message style ("u" for "you", "2" for "to").
Some spam is about adult services; it is real data.
6. What you must be able to explain
You may use AI for anything in this quest: to explain an idea, write code, or find mistakes. In the defense, the examiner asks about your code and your filter. Be ready to:
- work out a message's score by hand from its clues and your parameters;
- say which numbers are clues and which are parameters, and what changes when one parameter changes (for example a larger bias, or a higher threshold);
- explain how your
evaluatecounts tp, fp, fn and tn, and what it returns when nothing is flagged; - explain a mistake your filter makes, and why no weights can fix some mistakes;
- change your code, for example to report a new number.
What to submit
With neugen submit, from your quest folder:
filter.pyandmy_filter.json;- your AI transcripts or notes in
.neugen/logs/(neugen log <file>), if you used AI; SUBMISSION.md(the starter kit has a template), short: 1. My filter and why: your numbers, the reason for each weight, and two or three earlier attempts with their recall and precision. 2. Mistakes I looked at: two or three real messages your filter gets wrong, and what they tell you. 3. What a parameter is: in your own words, with one example from your filter. 4. How I used and checked AI: what you asked, and how you checked the answers.
How you're graded
- Automated checks (
neugen testruns the public ones, on the validation messages): - score, predict and evaluate (all must pass): your functions give the right answers on fixed inputs. The hidden checks include cases the public ones don't: ties at the threshold, negative weights, an empty message, no flagged messages, no spam. - Your filter, measured on the hidden test messages: it catches at least 30% of the spam and flags at most 20% of the normal messages. These gates are easy on purpose. - Rubric review by an AI grader: your reasons for the weights, how well you read your mistakes, and how you checked your work (and the AI's).
- Defense: a short conversation about your filter and your code.
Suggested path (about 4 hours)
- Read this brief and the CS231n sections (1 hour).
- Write
filter.py. Checkscoreagainst the worked example above by hand (45 minutes). - Choose weights, measure, read mistakes, repeat (1 hour 15 minutes).
- Write
SUBMISSION.md, runneugen test, submit (45 minutes).
Tips
- Check one score by hand before you trust
score(). Use message A, B or C above. - Change one number at a time, and write down the result. Otherwise you can't tell which change helped.
- Read the clues of a mistake, not only its text. If a spam message has no link, no prize words and few capitals, what score can any weights give it?
- Precision and recall pull against each other. Flag more messages and you catch more spam, but you also flag more normal ones. Decide which mistake is worse for the person using the filter.
Interface
# filter.py (clues come from features.extract(text): [has_link, prize_words, caps_fraction, length])
def score(clues: list[float], weights: list[float], bias: float) -> float: ...
# bias + the sum of weight * clue
def predict(clues, weights, bias, threshold=0.0) -> int: ...
# 1 (spam) if score > threshold, else 0 (normal). A tie is 0.
def evaluate(messages, weights, bias, threshold=0.0) -> dict: ...
# messages: a list of (text, label) pairs, as features.load_messages returns them.
# Uses features.extract. Returns at least:
# {"tp", "fp", "fn", "tn": counts, "accuracy", "precision", "recall": numbers}
# precision is 0.0 if nothing is flagged; recall is 0.0 if there is no spam.
# my_filter.json: the filter you chose by hand
{"weights": [4 numbers], "bias": number, "threshold": number}
Read now
Read only 'Parameterized mapping from images to label scores' and 'Interpreting a linear classifier'. Stop at 'Loss function' (that is the next quest). The notes use images and matrices; read every 'pixel' as one of your four clues, and W x + b as bias + the sum of weight * clue.
Explore further
Read 'Accuracy', 'Recall', 'False positive rate', 'Precision' and 'Choice of metric and tradeoffs'. Exactly the numbers evaluate() returns, with a spam example.
Three stages. You only move on when the one before passes.
Automated gates
| Metric | Gate | What it checks |
|---|---|---|
| score, predict and evaluate | ≥ 1.0 | Share of checks of your three functions that give the right answer. The hidden checks include ties at the threshold, negative weights, an empty message, no flagged messages and no spam. All must pass. |
| Spam caught by your filter | ≥ 0.3 | The share of spam messages that the filter in my_filter.json flags (its recall). A filter that never flags anything catches none. |
| Normal messages flagged by your filter | ≤ 0.2 | The share of normal messages that the filter in my_filter.json wrongly flags. A filter that flags everything flags all of them. |
Gates run on the public set (which you have) and a hidden set with inputs you haven't seen.
Rubric review need ≥ 60%
"My filter and why" gives the numbers in my_filter.json and a reason for each weight that refers to the clues or to real messages (e.g. "a link appears in normal order notices too, so it gets a smaller weight than prize words"). It shows at least two earlier attempts with their measured recall and precision on the validation messages and what changed. "What a parameter is" separates the numbers that describe a message (the clues) from the numbers that control the filter (weights, bias, threshold), with one concrete change and its effect.
"Mistakes I looked at" quotes two or three real misclassified messages from the data with their clues and score, and explains each one: which clue misled the filter or was missing, and whether better weights could fix it or the four clues simply do not contain the information (e.g. spam with no link and no prize words looks like a normal message). Good answers notice that a link alone does not mean spam, or that a score is not a probability.
The logs and "How I used and checked AI" show how the student checked what they did not write or work out themselves: e.g. worked out one score by hand and compared it with score(), tested evaluate() on three messages whose counts they knew, or asked the AI to explain precision and then checked it on their own numbers. Using AI to write the code is fine. No AI use is also fine if the student shows how they checked their own work.
Defense need ≥ 50%
A 4-question conversation with the AI examiner about your code. It isn't a trick: if you did it and understood it, it's the easy part. A weak defense on strong work gets you a new variant to redo.

