How Neural Networks Learn: Forward Propagation, Backpropagation & Gradient Descent

In the last lesson we saw what a neural network is — neurons in layers, connected by weights. But we skipped the most important question: a fresh network starts with completely random weights and knows nothing. So how does it go from random guessing to reading handwritten digits with 97% accuracy?

The answer is one elegant loop, repeated thousands of times. Once you understand it, the entire field of deep learning clicks into place. Let's walk through it — no calculus required.

Learning = guess, measure, adjust, repeat

Neural networks learn the same way you'd learn to throw darts blindfolded: throw, someone tells you how far off you were, you adjust, and throw again. Every "round" of training has four steps:

Let's unpack each step.

Step 1: Forward propagation (the guess)

This is exactly the flow from Lesson 15: feed the input in, let it travel through the layers — each neuron multiplying by weights and adding a bias — until the output layer produces a prediction. With random starting weights, this first guess is basically nonsense. That's fine; that's what the other three steps fix.

Step 2: The loss (how wrong were we?)

To improve, the network needs a score for how bad its guess was. That score is the loss (also called cost or error) — a single number produced by a loss function that compares the prediction to the true answer.

• Big loss = the prediction was far off.
• Small loss = the prediction was close.

The entire goal of training is simple to state: make the loss as small as possible.

Step 3 & 4: Gradient descent (rolling downhill)

Here's the beautiful part. Imagine plotting the loss against the network's weights. It forms a valley — some weight settings give high loss (up the slopes), the best settings give the lowest loss (the bottom). Learning means finding the bottom of that valley.

Gradient descent is the method for walking downhill. At the current spot it asks, "which way is downhill, and how steep?" (that's the gradient), then takes a small step in that direction. Do this over and over and you reach the bottom — the weights that give the lowest error.

Backpropagation is the clever algorithm that computes that downhill direction for every single weight at once, working backwards from the output error through each layer. It figures out how much each weight contributed to the mistake, so each one knows exactly which way to nudge. Then the weights update — and we loop back to step 1.

Backpropagation sends the error backwards to assign blame; gradient descent uses that blame to take a step downhill. Together they are how a network learns.

The learning rate and epochs

Two words you'll see everywhere:

The loop in code

Stripped to its essence, every deep learning framework is doing this each epoch:

for epoch in range(epochs):
    prediction = model.forward(X)              # 1. forward pass
    loss       = loss_fn(prediction, y_true)   # 2. how wrong?
    gradients  = loss.backward()               # 3. backpropagation
    weights   -= learning_rate * gradients     # 4. gradient descent step

You don't write this by hand in practice — libraries do it for you — but that's literally all that's happening inside. And you can watch it work. Scikit-learn's neural network records the loss after every epoch, and you'll see it fall as the network learns:

from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier

X, y = load_digits(return_X_y=True)
model = MLPClassifier(hidden_layer_sizes=(64,), max_iter=200)
model.fit(X, y)

# peek at the loss after each epoch — it should keep dropping
print([round(l, 3) for l in model.loss_curve_[:8]])
# e.g. [2.31, 1.74, 1.22, 0.86, 0.61, 0.44, 0.33, 0.26]  → learning!

That falling sequence is learning, made visible — each number is the network at the bottom of a slightly deeper point in the valley.

The moment gradient descent stopped being a scary term for me was when I first printed that loss curve and watched the numbers slide down epoch after epoch — I could finally see the network getting less wrong on its own, and it turned an abstract idea into something concrete I trusted.

Key takeaways

• Networks learn by looping four steps: forward pass → loss → backpropagate → update weights.
• The loss is a single number measuring how wrong a prediction is; training minimizes it.
• Gradient descent walks the weights "downhill" toward the lowest loss, one small step at a time.
• Backpropagation works backwards to tell each weight which way to move.
• The learning rate sets step size; an epoch is one full pass over the data.

You now understand the engine that trains every neural network on earth. But so far our networks are plain stacks of layers. Real deep learning uses specialised architectures — and the first big one, built for images, is the Convolutional Neural Network (CNN). That's next.

Continue the series: ← Lesson 15: What is a Neural Network?  ·  Next: Lesson 17 — Convolutional Neural Networks (CNNs) →