Accuracy, Precision, Recall & the Confusion Matrix: How to Measure a Machine Learning Model

In the last lesson we trained four classification algorithms and said "keep the winner." But how do you actually know which one won? A model always spits out predictions — the real question is whether those predictions are any good.

This is where a lot of beginners trip up. They check one number — "accuracy: 95%!" — celebrate, and ship a model that's secretly useless. This lesson shows you why, and gives you the small set of metrics that professionals actually rely on: the confusion matrix, accuracy, precision, recall, and F1 score.

Why accuracy alone can lie to you

Imagine you build a model to detect a rare disease that affects 1 in 100 people. You write a "model" that is just one line: always predict "healthy."

That lazy model is 99% accurate — it's right about the 99 healthy people and only wrong about the 1 sick person. Sounds great! Except it never catches a single sick patient. It's completely worthless, yet accuracy calls it excellent.

This is the imbalanced data trap, and it's everywhere: fraud detection, spam, disease screening, defect detection. Whenever one class is rare, accuracy stops being trustworthy. We need to look deeper — starting with the confusion matrix.

The confusion matrix

Every prediction a binary classifier makes falls into one of four buckets. Lay them out in a 2×2 grid and you get the confusion matrix — the foundation every other metric is built on.

Read each term as two words — was the model right (True) or wrong (False), and what did it say (Positive/Negative):

• True Positive (TP): predicted "sick", the person really is sick. ✓
• True Negative (TN): predicted "healthy", the person really is healthy. ✓
• False Positive (FP): predicted "sick", but they were healthy — a false alarm. ✗
• False Negative (FN): predicted "healthy", but they were actually sick — a dangerous miss. ✗

Notice the two mistakes are not equally bad. A false positive means an unnecessary follow-up test. A false negative means a sick patient walks away untreated. Which one matters more depends entirely on the problem — and that's exactly what precision and recall let you control.

Accuracy

Accuracy = of all predictions, how many were right?

# the greens, over everything
Accuracy = (TP + TN) / (TP + TN + FP + FN)

It's a fine first glance, and perfectly good when your classes are roughly balanced. But as the disease example showed, on imbalanced data it hides the failures. So don't stop here.

Precision

Precision = when the model says "positive", how often is it right?

# of everything flagged positive, how much was truly positive
Precision = TP / (TP + FP)

Precision punishes false alarms. Ask for it when a false positive is expensive. A spam filter is the classic case: if it wrongly flags an important email as spam (a false positive) you might miss a job offer — so you want high precision. Better to let a little spam through than to lose real mail.

Recall

Recall = of all the actual positives, how many did the model catch?

# of all the real positives, how many did we find
Recall = TP / (TP + FN)

Recall punishes misses. Ask for it when a false negative is dangerous. Disease screening is the classic case: missing a sick patient (a false negative) can cost a life, so you want high recall even if it means a few more false alarms. Better to call in a healthy person for a second test than to send a sick one home.

The precision–recall trade-off

Here's the catch: precision and recall usually pull against each other. Push the model to catch every possible positive (high recall) and it starts flagging borderline cases, creating false alarms (lower precision). Make it flag only the cases it's very sure about (high precision) and it misses the subtle ones (lower recall).

You can't max both — you choose which to favour based on the cost of each mistake. That single decision, "which error hurts more here?", is one of the most important judgment calls in applied machine learning. [Optional: add a line about a time you had to make this call on a real project, so it reflects your own experience.]

F1 score

When you care about precision and recall and want one number to compare models, use the F1 score — the balanced (harmonic) mean of the two:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

The harmonic mean only stays high when both numbers are high — if either precision or recall is poor, F1 drops. That makes F1 a great single scorecard for imbalanced problems, where plain accuracy would mislead you.

You don't compute these by hand

scikit-learn gives you every metric above in a couple of lines. In practice this is all you write:

from sklearn.metrics import confusion_matrix, classification_report

# y_test = true labels, y_pred = model's predictions
print(confusion_matrix(y_test, y_pred))

# accuracy, precision, recall and F1 for every class, in one table
print(classification_report(y_test, y_pred))

The classification_report is the one to memorise — it prints precision, recall, F1 and support for each class at once, so you can read the full story instead of a single misleading number.

Which metric should you focus on?

Key takeaways

• Accuracy alone can lie — on imbalanced data a useless model can still look 99% accurate.
• The confusion matrix splits predictions into TP, TN, FP, FN — the base for every other metric.
• Precision guards against false alarms; recall guards against misses; they trade off against each other.
• F1 score balances both into one number — ideal for imbalanced problems.
• In code, classification_report() gives you all of it at once.

Now you can train several models and judge them honestly. Next we'll tackle a problem hiding behind every "great" score: overfitting vs underfitting — why a model that scores 100% on training data can still fail in the real world, and how train/test splits and cross-validation protect you.

Continue the series: ← Lesson 10: Classification Algorithms · Lesson 12: Overfitting, Underfitting & Cross-Validation (coming next).