Learn
Learning 🤖 AI-on 💻 On your computer ⏱ ~4h Difficulty 1/5 v1

Build a message filter

Unwanted messages are annoying and sometimes dangerous. Build a filter that gives each message a score from a few clues (a link, prize words, capital letters, length), choose its weights by hand, measure how well it works and read the messages it gets wrong. Your first model: a function whose behaviour is set by numbers you choose.

Join free to start → Read the brief and materials. Join AI from Scratch to start, submit and ask the tutor. Log in

Why this quest

Your phone receives three messages:

Message
A Congratulations! You have won a £500 prize. To claim, call 09061701461 before midnight. Ref: KL341
B Your order 4471 has shipped. Track it at www.parcelhub.example.com/track. Expected delivery: Thursday.
C Hi Ana, can we move tomorrow's meeting to 3pm? Same room. Thanks, Kenji

You know at once that A is unwanted (spam) and that B and C are normal. But how do you know? Perhaps the prize, the money sign, the urgent "before midnight". B has a link, like a lot of spam, but it is still a normal message. A computer needs numbers and a rule.

In this quest you build that rule: a small message filter. It looks at a few clues about a message, gives the message a score, and flags it as spam when the score is high enough. You choose the filter's numbers by hand, measure how well it works on real messages, and read the messages it gets wrong. This is your first model: a function whose behaviour is set by numbers you choose. In the next quest, the computer chooses those numbers for you.

1. Clues: numbers that describe a message

The starter kit's features.py turns a message into four numbers, called clues (or features):

Clue Meaning A B C
has_link 1 if it contains a web address, else 0 0 1 0
prize_words how many prize or money words (free, win, prize, cash, claim, £...) 5 0 0
caps_fraction share of its letters that are capitals (0 to 1) 0.0984 0.0506 0.0943
length its length in characters, divided by 100 0.98 1.02 0.71
from features import extract
extract("Congratulations! You have won a £500 prize. ...")   # -> [0.0, 5.0, 0.0984, 0.98]

You don't need to change features.py. The grader uses its own copy.

2. Parameters: numbers that control the filter

Here is a question to keep in mind for the whole quest: which numbers describe the message, and which numbers control the filter?

The clues describe the message. You can't change them. The filter has its own numbers, which you choose: one weight per clue and one bias. These are its parameters.

The filter's score for a message is:

score = bias + weight1 * has_link + weight2 * prize_words + weight3 * caps_fraction + weight4 * length

A worked example. Take weights [2, 1.5, 3, 0.5] and bias -2:

A: -2 + 2*0 + 1.5*5 + 3*0.0984 + 0.5*0.98 = -2 + 0 + 7.5 + 0.2952 + 0.49  =  6.2852
B: -2 + 2*1 + 1.5*0 + 3*0.0506 + 0.5*1.02 = -2 + 2 + 0   + 0.1518 + 0.51  =  0.6618
C: -2 + 2*0 + 1.5*0 + 3*0.0943 + 0.5*0.71 = -2 + 0 + 0   + 0.2829 + 0.355 = -1.3621

A big positive weight means "this clue makes a message more suspicious". A weight of 0 means "ignore this clue". A negative weight means "this clue makes a message look more normal". The bias is the starting point before any clue is counted: a negative bias means "a message is normal until the clues say otherwise".

To decide, compare the score with a threshold. If the score is greater than the threshold, the filter predicts 1 (spam); otherwise it predicts 0 (normal). With threshold 0, this filter flags A (6.29) and leaves C (-1.36). But it also flags B (0.66), the order notice. Which number would you change to fix B, and what might that break? That is the work of this quest.

Note: a score is not a probability. A score of 6.29 does not mean "629% sure". It only says "more suspicious than a message with a lower score". (The next quest turns scores into probabilities.)

3. Measuring a filter

Every message in the data has a label: 1 for spam, 0 for normal. People wrote these labels, not the filter. Compare each prediction with its label, and count four kinds of result:

the filter says spam (1) the filter says normal (0)
really spam true positive (tp) false negative (fn): missed
really normal false positive (fp): a false alarm true negative (tn)

From these counts:

  • accuracy = (tp + tn) / all messages: the share it gets right.
  • precision = tp / (tp + fp): of the messages it flags, the share that really are spam.
  • recall = tp / (tp + fn): of the spam messages, the share it catches.

A small example: 10 messages, 2 of them spam. The filter flags 3 messages: 1 spam and 2 normal. Then tp = 1, fp = 2, fn = 1, tn = 6. Accuracy = 7/10 = 0.7, precision = 1/3 ≈ 0.33, recall = 1/2 = 0.5.

Why three numbers? About 86% of the messages in this data are normal. A "filter" that never flags anything has 86% accuracy, and catches no spam at all: recall 0. Always look at recall and precision, not only accuracy.

When nothing is flagged, precision would divide by zero. In this quest, precision is 0.0 when nothing is flagged, and recall is 0.0 when there is no spam in the list.

4. What to build

Two files, in your starter kit:

filter.py, with three functions (the docstrings explain each one):

def score(clues, weights, bias) -> float            # bias + the sum of weight * clue
def predict(clues, weights, bias, threshold=0.0) -> int   # 1 if score > threshold, else 0
def evaluate(messages, weights, bias, threshold=0.0) -> dict

messages is a list of (text, label) pairs, as features.load_messages("data/sms_val.tsv") returns them. evaluate gets each message's clues with features.extract(text) and returns a dict with at least tp, fp, fn, tn, accuracy, precision and recall. Keep these names exactly: the next quests use them.

my_filter.json: the filter you choose by hand.

{"weights": [2, 1.5, 3, 0.5], "bias": -2, "threshold": 0}

Then choose your parameters. Read some spam and normal messages in data/sms_train.tsv. Think about each clue: how often does it appear in spam, and in normal messages? Pick weights, measure, read the mistakes, change one number, measure again. python try_filter.py prints the numbers and some mistakes on the validation messages. Keep notes of what you tried: you will need them for your write-up.

You don't need a perfect filter. The goal is to understand what each number does. A thoughtful filter catches well over half of the spam on these four clues; no choice of weights catches all of it. Find out why.

5. The data

data/ holds real text messages from the public SMS Spam Collection (collected in 2011, mostly from the UK and Singapore; see data/SOURCE.md):

  • sms_train.tsv (3,900 messages): read these and use them to choose your weights.
  • sms_val.tsv (833 messages): check your filter on these (try_filter.py uses them).
  • A hidden test set (841 messages) that you never see. The grader measures your filter on it.

Each line is label<TAB>text. Messages use 2011 text-message style ("u" for "you", "2" for "to"). Some spam is about adult services; it is real data.

6. What you must be able to explain

You may use AI for anything in this quest: to explain an idea, write code, or find mistakes. In the defense, the examiner asks about your code and your filter. Be ready to:

  • work out a message's score by hand from its clues and your parameters;
  • say which numbers are clues and which are parameters, and what changes when one parameter changes (for example a larger bias, or a higher threshold);
  • explain how your evaluate counts tp, fp, fn and tn, and what it returns when nothing is flagged;
  • explain a mistake your filter makes, and why no weights can fix some mistakes;
  • change your code, for example to report a new number.

What to submit

With neugen submit, from your quest folder:

  • filter.py and my_filter.json;
  • your AI transcripts or notes in .neugen/logs/ (neugen log <file>), if you used AI;
  • SUBMISSION.md (the starter kit has a template), short: 1. My filter and why: your numbers, the reason for each weight, and two or three earlier attempts with their recall and precision. 2. Mistakes I looked at: two or three real messages your filter gets wrong, and what they tell you. 3. What a parameter is: in your own words, with one example from your filter. 4. How I used and checked AI: what you asked, and how you checked the answers.

How you're graded

  1. Automated checks (neugen test runs the public ones, on the validation messages): - score, predict and evaluate (all must pass): your functions give the right answers on fixed inputs. The hidden checks include cases the public ones don't: ties at the threshold, negative weights, an empty message, no flagged messages, no spam. - Your filter, measured on the hidden test messages: it catches at least 30% of the spam and flags at most 20% of the normal messages. These gates are easy on purpose.
  2. Rubric review by an AI grader: your reasons for the weights, how well you read your mistakes, and how you checked your work (and the AI's).
  3. Defense: a short conversation about your filter and your code.

Suggested path (about 4 hours)

  1. Read this brief and the CS231n sections (1 hour).
  2. Write filter.py. Check score against the worked example above by hand (45 minutes).
  3. Choose weights, measure, read mistakes, repeat (1 hour 15 minutes).
  4. Write SUBMISSION.md, run neugen test, submit (45 minutes).

Tips

  • Check one score by hand before you trust score(). Use message A, B or C above.
  • Change one number at a time, and write down the result. Otherwise you can't tell which change helped.
  • Read the clues of a mistake, not only its text. If a spam message has no link, no prize words and few capitals, what score can any weights give it?
  • Precision and recall pull against each other. Flag more messages and you catch more spam, but you also flag more normal ones. Decide which mistake is worse for the person using the filter.

Interface

# filter.py   (clues come from features.extract(text): [has_link, prize_words, caps_fraction, length])
def score(clues: list[float], weights: list[float], bias: float) -> float: ...
    # bias + the sum of weight * clue
def predict(clues, weights, bias, threshold=0.0) -> int: ...
    # 1 (spam) if score > threshold, else 0 (normal). A tie is 0.
def evaluate(messages, weights, bias, threshold=0.0) -> dict: ...
    # messages: a list of (text, label) pairs, as features.load_messages returns them.
    # Uses features.extract. Returns at least:
    # {"tp", "fp", "fn", "tn": counts, "accuracy", "precision", "recall": numbers}
    # precision is 0.0 if nothing is flagged; recall is 0.0 if there is no spam.

# my_filter.json: the filter you chose by hand
{"weights": [4 numbers], "bias": number, "threshold": number}