Recurrent Neural Networks (RNNs) and LSTMs Explained

In Lesson 17 you saw how CNNs conquer images. But a huge amount of data isn't a single snapshot — it's a sequence: the words in this sentence, a week of stock prices, the sound of speech. In sequences, order carries meaning. "Dog bites man" and "man bites dog" use the same words but mean opposite things. Recurrent Neural Networks (RNNs) are the architecture built to handle order and context — and LSTMs are the upgrade that gave them a proper memory. This lesson makes both intuitive.

Why sequences need a different network

The networks from earlier lessons are feed-forward: data goes in one end, a prediction comes out the other, and each input is treated independently. That's fine for a photo. But to understand the word "it" in a sentence, the network must remember what came before. A CNN or a plain network has no memory of the previous input — every prediction starts from a blank slate.

RNNs add exactly one thing: a memory that carries forward.

The core idea: a loop with memory

An RNN processes a sequence one item at a time. At each step it combines two things — the current input and a summary of everything it has seen so far (called the hidden state). It produces an output and an updated hidden state, then passes that state to the next step. The same network cell is reused at every step; it just keeps receiving its own memory back.

It's easiest to picture "unrolled" across time:

It's really one cell used repeatedly — the diagram just stretches time out left to right so you can see the memory flowing.

See one RNN step in NumPy (runnable)

The whole "memory" idea is one short formula: the new hidden state is a blend of the old hidden state and the new input, squished by tanh. This runs with just NumPy — watch the hidden state change as each word arrives.

import numpy as np
np.random.seed(0)

# Pretend each word is a 3-number vector
sequence = {
    "I":    np.array([1.0, 0.0, 0.0]),
    "love": np.array([0.0, 1.0, 0.0]),
    "AI":   np.array([0.0, 0.0, 1.0]),
}

# Weights (learned in a real RNN; random here to demo)
Wx = np.random.randn(4, 3) * 0.5   # input -> hidden
Wh = np.random.randn(4, 4) * 0.5   # memory -> hidden

h = np.zeros(4)                    # start with empty memory
for word in ["I", "love", "AI"]:
    x = sequence[word]
    h = np.tanh(Wx @ x + Wh @ h)   # blend input + old memory
    print(f"after '{word}':", np.round(h, 2))

# The final h is a summary of the WHOLE sentence.

Notice the hidden state after "AI" depends on "I" and "love" too — that's memory. In a trained RNN those weights are learned so the final summary is genuinely useful for a task like sentiment or translation.

The catch: RNNs have a short memory

Plain RNNs work for short sequences but forget things over long ones. Because the same weights are applied again and again, the signal from early words shrinks (or explodes) as it passes through many steps — the vanishing gradient problem from Lesson 16, made worse by depth in time. Ask a plain RNN to connect "The cat, which I adopted last winter during a snowstorm, was hungry" and by the time it reaches "was" it has often forgotten "cat."

LSTMs: giving the network gates

A Long Short-Term Memory (LSTM) network fixes this with a smarter cell. Alongside the hidden state it keeps a separate cell state — a conveyor belt that carries information across many steps with little change. Small neural "gates" decide what to do at each step:

Because the cell state flows along that conveyor belt with only gentle, gated edits, an LSTM can hold onto "cat" long enough to correctly pick "was." A close cousin, the GRU, does the same job with fewer gates and is a bit faster — you'll see both names in practice.

What real code looks like

You never wire up gates by hand — Keras or PyTorch give you an LSTM in one line. Here's a small model that reads a sequence and makes one prediction (e.g. positive/negative review). (Needs pip install tensorflow; read it to recognise the shape.)

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Embedding(input_dim=10000, output_dim=32),  # words -> vectors
    layers.LSTM(64),                 # the recurrent memory layer
    layers.Dense(1, activation="sigmoid")  # positive / negative
])

model.compile(optimizer="adam",
              loss="binary_crossentropy",
              metrics=["accuracy"])
model.summary()

Swap LSTM for SimpleRNN or GRU and the rest is identical — that's how interchangeable these layers are.

Where RNNs and LSTMs are used

One line to remember: a plain RNN has memory but a leaky one; an LSTM adds gates so it can choose what to keep and what to forget — memory with a manager.

When I first swapped a SimpleRNN for an LSTM on a small text task and watched the accuracy on long reviews jump, the whole "gates give it real memory" idea stopped being abstract for me. If you're experimenting, try both on the same dataset — feeling the difference yourself is worth more than any explanation.

Key takeaways

• RNNs are for sequences (text, audio, time series) where order matters; CNNs are for images.
• An RNN reuses one cell across time, passing a hidden state forward as memory.
• Plain RNNs forget long-range information (vanishing gradients).
• LSTMs add a cell-state conveyor and forget / input / output gates to keep important information for longer; GRUs are a lighter variant.
• In code they're one interchangeable layer in Keras/PyTorch — but they trained the same way you learned in Lesson 16.

Continue the series: ← Lesson 17: Convolutional Neural Networks (CNNs)  ·  Next: Lesson 19 — Transformers and Attention (the tech behind ChatGPT) →