Classification Algorithms in Machine Learning: KNN, Decision Tree, Random Forest & SVM

In the last lesson we looked at linear and logistic regression, where logistic regression let us predict a category like "yes or no". But logistic regression is just one way to classify things. In real projects you'll reach for a handful of other algorithms all the time — and each one draws the line between classes in its own way.

This lesson covers the four classification algorithms you'll use most as a beginner: K-Nearest Neighbors (KNN), Decision Tree, Random Forest, and Support Vector Machine (SVM). We'll keep it plain-English, use one running example, and give you copy-paste Python for each.

First: what is "classification"?

Classification means predicting which category something belongs to. The answer is a label, not a number:

• Is this email spam or not spam?
• Is this tumour benign or malignant?
• Is this customer going to churn or stay?
• Which of 3 flower species is this?

When there are two labels it's binary classification; more than two is multi-class. All four algorithms below handle both.

The key idea to hold onto: every classifier is really just drawing a boundary through your data. Points on one side get one label, points on the other side get another. What makes the algorithms different is the shape of that boundary and how they decide where to draw it.

1. K-Nearest Neighbors (KNN)

The idea in one line: to classify a new point, look at its closest neighbours and go with the majority.

KNN doesn't really "learn" anything upfront. It just remembers all the training data. When a new point arrives, it measures the distance to every stored point, picks the K closest ones (K might be 3, 5, 7…), and lets them vote. If 4 of the 5 nearest points are "spam", the new email is spam.

It's the friend-group heuristic: "you're most like the people standing closest to you." In the diagram above, the purple point becomes blue because its neighbours inside the dashed circle are mostly blue.

from sklearn.neighbors import KNeighborsClassifier

# look at the 5 nearest neighbours
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

predictions = model.predict(X_test)

• Good for: small datasets, simple problems, when the boundary is irregular.
• Watch out: slow on big data (it compares against every point), and you must scale your features first (distance is meaningless if one column is in "thousands" and another in "0–1").
• Pick K carefully: too small = noisy; too large = it ignores the local pattern. An odd number avoids tie votes.

2. Decision Tree

The idea in one line: ask a series of yes/no questions until you can name the answer.

A decision tree splits the data with simple threshold questions — "Is age > 30?", "Is income > 50k?" — and keeps splitting until each branch is mostly one class. Predicting is just walking down the branches. It mirrors how a human would reason through a problem, which is why trees are the easiest model to explain to a non-technical person.

In the diagram, notice the boundary is made of straight horizontal/vertical cuts — that's exactly what threshold questions produce.

from sklearn.tree import DecisionTreeClassifier

# max_depth stops the tree from growing too complex
model = DecisionTreeClassifier(max_depth=4)
model.fit(X_train, y_train)

predictions = model.predict(X_test)

• Good for: when you need an explainable model, mixed data types, no feature scaling needed.
• Watch out: a single deep tree overfits easily — it memorises the training data and does poorly on new data. Limiting max_depth helps. This weakness is exactly what the next algorithm fixes.

3. Random Forest

The idea in one line: build hundreds of decision trees and let them vote.

One tree is unstable. But if you grow many trees, each on a slightly different random slice of the data and features, their mistakes cancel out and the majority vote is far more reliable. That's a Random Forest — a whole "forest" of trees.

This is called an ensemble (many models combined). It's one of the most reliable, best "just works" algorithms for everyday tabular data — often my first choice before reaching for anything fancier. [Rewrite this line in your own words — e.g. mention the actual project where Random Forest gave you a solid baseline, so it reflects your real experience.]

from sklearn.ensemble import RandomForestClassifier

# 100 trees voting together
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

predictions = model.predict(X_test)

• Good for: most real-world tabular problems — a strong, low-effort baseline. Handles overfitting well.
• Watch out: less explainable than a single tree, and heavier to run. But it also tells you which features mattered most (model.feature_importances_), which is genuinely useful.

4. Support Vector Machine (SVM)

The idea in one line: draw the boundary that leaves the widest possible gap between the two classes.

Where other algorithms are happy with any line that separates the classes, SVM insists on the best line — the one that sits as far as possible from the closest points of each class (those closest points are the "support vectors"). That wide margin, shown by the dashed lines in the diagram, tends to generalise well to new data.

SVM has one more trick: with a "kernel" it can bend the boundary into curves to separate classes that a straight line never could. This makes it powerful for complex, high-dimensional data like text.

from sklearn.svm import SVC

# 'rbf' kernel lets the boundary curve
model = SVC(kernel='rbf')
model.fit(X_train, y_train)

predictions = model.predict(X_test)

• Good for: clear-margin problems, high-dimensional data (e.g. text classification), medium-sized datasets.
• Watch out: slow on very large datasets, sensitive to feature scaling, and harder to tune. Not the friendliest starting point.

Quick comparison

So which one should you use?

You don't have to guess perfectly — in practice you try a couple and compare. But a sensible default order for a beginner:

1. Start with Random Forest. It works well out of the box on most tabular data and needs little tuning.
2. Try a Decision Tree if you need to explain the model to someone.
3. Try SVM for high-dimensional problems like text.
4. Try KNN for small, simple datasets or as an easy baseline.

The beautiful part: because scikit-learn gives every model the same .fit() and .predict() interface, swapping one for another is a one-line change. Train all four, compare their accuracy, keep the winner.

Key takeaways

• Classification predicts a category (label), and every classifier is really just drawing a boundary between classes.
• KNN votes with nearest neighbours; Decision Tree asks yes/no questions; Random Forest is many trees voting; SVM finds the widest-margin boundary.
• Random Forest is the best all-round default for beginners on tabular data.
• Thanks to scikit-learn's common interface, trying and comparing them is only a one-line change.

Next up, we'll look at how to actually measure whether your classifier is any good — accuracy, precision, recall and the confusion matrix — because "it predicted something" isn't the same as "it predicted correctly".

Continue the series: ← Lesson 9: Linear & Logistic Regression · Lesson 11: Evaluating a Model (coming next).