RAG Explained: Give AI Your Own Data and Fewer Hallucinations

In Lesson 20 you learned two limits of every LLM: its knowledge is frozen at training time, and it can hallucinate confident nonsense. So how do tools like "chat with your PDF," AI customer-support bots, and internal company assistants give accurate, up-to-date answers about your documents? The answer is RAG — Retrieval-Augmented Generation. It's one of the most useful and in-demand techniques in applied AI, and the idea is beautifully simple.

The problem: an LLM only knows what it was trained on

Ask a plain LLM about your company's refund policy, a document you wrote yesterday, or last week's news, and it either doesn't know or invents an answer. You can't retrain a billion-parameter model every time a fact changes — that's slow and expensive. You need a way to feed it the right information at question time.

The idea: let the AI take an open-book exam

Think of a plain LLM as a student answering from memory — impressive, but it forgets and bluffs. RAG turns it into an open-book exam: before the model answers, we look up the most relevant passages from your documents and hand them to the model along with the question. Now it answers from the text in front of it, not from fuzzy memory. Accuracy jumps, and hallucinations drop.

How retrieval finds the right passage: embeddings

How does the retriever know which passages are "relevant"? Keyword matching is too brittle — "car" wouldn't match "automobile." Instead RAG uses embeddings: a model turns each piece of text into a list of numbers (a vector) that captures its meaning. Texts with similar meaning end up as nearby vectors. To find relevant passages, we embed the question and search for the document vectors closest to it. That search lives in a vector database (like FAISS, Pinecone or Chroma).

See retrieval in NumPy (runnable)

Here's the core of a retriever with no libraries — we turn short texts into simple word-count vectors and use cosine similarity to find the passage closest in meaning to the question. Run it: the most relevant sentence is retrieved automatically.

import numpy as np

docs = [
    "Our refund window is 30 days from purchase.",
    "The office is open Monday to Friday, 9 to 5.",
    "Returns require the original receipt and packaging.",
]
question = "How many days do I have to get a refund?"

# Tiny 'embedding': count shared vocabulary words
vocab = sorted(set(" ".join(docs + [question]).lower().replace(".", "").split()))
def embed(text):
    words = text.lower().replace("?", "").replace(".", "").split()
    return np.array([words.count(w) for w in vocab], dtype=float)

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9)

q = embed(question)
scores = [cosine(q, embed(d)) for d in docs]
best = int(np.argmax(scores))

print("Retrieved:", docs[best])   # -> the refund-window sentence
# That chunk is then pasted into the LLM prompt.

Real systems use smarter embeddings (from a neural network, so "refund" and "money back" match too), but the mechanism is exactly this: embed, compare, retrieve the closest, feed it to the LLM.

Chunking: splitting your documents

You don't embed a whole 50-page PDF as one vector — you'd retrieve too much irrelevant text. Documents are split into small chunks (say a few paragraphs each), and each chunk is embedded separately. At question time you pull only the handful of chunks that actually match. Good chunking is one of the biggest levers on RAG quality.

RAG vs fine-tuning: which do you need?

They're not rivals — many production systems do both. But for "answer accurately from my documents," RAG is almost always the right first choice.

Where RAG is used

• Chat-with-your-documents tools (PDFs, contracts, research papers).
• Customer support bots grounded in a help centre, so answers are accurate and cite sources.
• Internal company assistants that answer from private wikis and policies.
• Search + summarise over a knowledge base instead of endless scrolling.

One line to remember: RAG doesn't make the model smarter — it makes it informed, by handing it the right passage before it answers.

RAG was the technique that turned LLMs from "fun to chat with" into "actually trustworthy for real work" for me — the first time I built a tiny chat-with-my-notes tool and watched it quote my own document back to me instead of guessing, the practical value clicked instantly. If you try one AI project after this series, make it a small RAG app; it teaches you more than any article can.

Key takeaways

• RAG (Retrieval-Augmented Generation) lets an LLM answer from your own, current documents instead of only its frozen training data.
• It works like an open-book exam: retrieve relevant passages, then let the model answer using them.
• Retrieval uses embeddings (meaning-vectors) and cosine similarity in a vector database.
• Documents are split into chunks; only the matching chunks go into the prompt.
• Use RAG for knowledge, fine-tuning for style/skill — and RAG meaningfully reduces hallucinations.

Continue the series: ← Lesson 21: Prompt Engineering  ·  Next: Lesson 23 — AI Agents: When AI Takes Actions on Its Own →