Overfitting, Underfitting & Cross-Validation in Machine Learning (Beginner's Guide)

In the last lesson we learned how to measure a model with accuracy, precision and recall. But there's a trap those numbers can hide: a model can score a perfect 100% on the data it was trained on and still be useless on anything new.

That gap — between "looks great in training" and "works in the real world" — is the single most important idea separating a beginner from someone who ships models that actually work. It comes down to two failure modes, overfitting and underfitting, and the tools that protect you from them: the train/test split and cross-validation.

An analogy: studying for an exam

Imagine three students preparing for a maths exam:

• Student A (underfitting) barely studies and only learns "there will be some numbers." They do badly on practice questions and the real exam. They didn't learn enough.
• Student B (overfitting) memorises every practice question's answer word-for-word without understanding the concepts. They get 100% on the practice paper — but the moment the real exam changes the numbers, they're lost. They memorised instead of learning.
• Student C (good fit) actually understands the concepts. They do well on practice questions and on the real exam, even though the questions are new.

A machine learning model is exactly like these students. The "practice paper" is your training data; the "real exam" is new, unseen data. Our whole goal is to build Student C.

What the three look like

Picture the same scattered data points fit three different ways. The pattern is unmistakable once you've seen it:

Underfitting (too simple)

Underfitting happens when your model is too simple to capture the real pattern in the data — like drawing a straight line through data that clearly curves. It does poorly on the training data and on new data.

Signs: low training accuracy and low test accuracy — both bad, and close together.

Fixes: use a more powerful model, add more useful features, or train longer. The model needs more capacity to learn.

Overfitting (too complex)

Overfitting is the more common and more sneaky problem. The model is so flexible it memorises the training data — including the random noise that won't repeat — instead of learning the general pattern. It looks brilliant in training and falls apart on new data.

Signs: very high training accuracy but noticeably lower test accuracy — a big gap between the two. That gap is the tell-tale sign of overfitting.

Fixes: use a simpler model, get more training data, remove noisy features, or apply regularization (a technique that penalises overly complex models — we'll meet it later in the series).

Bias vs variance (the same idea, named)

You'll hear these two words constantly, and they're just the technical names for what we've described:

The bias–variance trade-off is the balancing act at the heart of machine learning: reduce bias too much and variance creeps up, and vice-versa. The sweet spot in the middle is our "good fit" — Student C.

The one habit that saves you: the train/test split

Here's the crucial rule that catches overfitting: never judge a model on the data it trained on. Just like you don't test a student with the exact questions they practised.

So we split our data — commonly 80% for training, 20% for testing. The model learns only from the training set, then we measure it on the test set it has never seen. That test score is our honest estimate of real-world performance.

from sklearn.model_selection import train_test_split

# keep 20% aside for honest testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

model.fit(X_train, y_train)          # learn on training only
print(model.score(X_test, y_test))   # judge on unseen test

If training score is high but test score is much lower — you're overfitting. If both are low — you're underfitting. This one comparison tells you almost everything.

A more reliable check: cross-validation

A single train/test split has a weakness: maybe you got lucky (or unlucky) with which 20% landed in the test set. Cross-validation removes that luck.

The most common form, k-fold cross-validation, splits the data into k equal parts (say 5). It then trains the model 5 times — each time holding out a different fifth as the test set and training on the other four. You average the 5 scores for a much more trustworthy estimate.

from sklearn.model_selection import cross_val_score

# train & test 5 times, get 5 scores
scores = cross_val_score(model, X, y, cv=5)

print(scores)              # e.g. [0.94 0.91 0.93 0.90 0.92]
print(scores.mean())       # the reliable single number

I lean on cross-validation whenever the dataset is small, where one unlucky split could really skew the picture. [Rewrite in your own words if you'd like — e.g. name a project where a single split gave a misleadingly high score.]

The first time I saw a model hit 100% on training data I was thrilled — until the test score came back far lower. That gap is what taught me to never trust a training score on its own.

Key takeaways

• Underfitting = model too simple; bad on training and test data (high bias).
• Overfitting = model memorises the training data including noise; great on training, poor on new data (high variance).
• The tell-tale sign of overfitting is a big gap between a high training score and a lower test score.
• Always use a train/test split — never judge a model on data it trained on.
• Cross-validation averages several splits for a far more reliable estimate, especially on small data.

Continue the series: ← Lesson 11: Accuracy, Precision & Recall  ·  Next: Lesson 13 — Data Preprocessing →