Build a RAG App Step by Step: A Hands-On Project

In Lesson 22 you learned how RAG works. Now let's build one. By the end of this lesson you'll have a working "chat with your documents" app in plain Python — the exact kind of project that stands out in a portfolio (Lesson 26). We'll keep the core runnable with just NumPy so you can follow along today, then show exactly what to swap in to make it production-grade. Let's write some code.

What we're building

A small system that answers questions using your text, not the model's general knowledge. Ask "how many days for a refund?" and it finds the relevant sentence in your documents and answers from it — grounded, not guessed. Six steps:

Step 1 & 2: Documents and chunking

Real apps load PDFs or web pages; we'll use a small text blob and split it into chunks (one sentence each here, for simplicity).

import numpy as np

knowledge = """GyaanPost is a blog for learning AI from scratch.
Our refund window is 30 days from the date of purchase.
The office is open Monday to Friday, 9am to 5pm.
Returns require the original receipt and packaging."""

# Chunk: one sentence (line) per chunk
chunks = [line.strip() for line in knowledge.split("\n") if line.strip()]
print(f"{len(chunks)} chunks ready")

Step 3 & 4: Embed the chunks and store them

We convert each chunk into a vector. Here we use a simple word-count embedding so it runs with no downloads; the "store" is just a NumPy array. (A real app uses a neural embedding model and a vector database — we'll swap those in later.)

vocab = sorted(set(knowledge.lower().replace(".", "").replace(",", "").split()))

def embed(text):
    words = text.lower().replace("?", "").replace(".", "").replace(",", "").split()
    return np.array([words.count(w) for w in vocab], dtype=float)

# Our 'vector store': every chunk embedded once, up front
store = np.array([embed(c) for c in chunks])
print("store shape:", store.shape)   # (4 chunks, vocab-size)

Step 5: Retrieve the most relevant chunks

Given a question, embed it and find the closest chunks by cosine similarity — the retriever's whole job.

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9)

def retrieve(question, k=2):
    q = embed(question)
    scores = [cosine(q, v) for v in store]
    top = np.argsort(scores)[::-1][:k]      # best k chunks
    return [chunks[i] for i in top]

print(retrieve("How many days do I have for a refund?"))
# -> ['Our refund window is 30 days...', ...] 

Step 6: Build the prompt and generate the answer

We stuff the retrieved chunks into a prompt that tells the LLM to answer only from that context — the key to grounding and fewer hallucinations. The call_llm function is a stub so the app runs end to end; the comment shows the real call.

def build_prompt(question, context):
    joined = "\n".join(f"- {c}" for c in context)
    return f"""Answer using ONLY the context below.
If the answer isn't there, say you don't know.

Context:
{joined}

Question: {question}
Answer:"""

def call_llm(prompt):
    # STUB — replace with a real LLM API call, e.g.:
    #   from openai import OpenAI
    #   client = OpenAI()
    #   return client.responses.create(model="...", input=prompt).output_text
    return "[The LLM answers here, grounded in the context above]"

Putting it all together

Now the whole RAG app is one function — retrieve, build, generate — and it returns the answer plus its sources (great for trust):

def ask(question):
    context = retrieve(question)              # step 5
    prompt  = build_prompt(question, context) # step 6a
    answer  = call_llm(prompt)                # step 6b
    return answer, context

answer, sources = ask("When can I get a refund?")
print("Answer :", answer)
print("Sources:", sources)   # the exact chunks used — cite them!

That's a complete RAG pipeline. The retrieval genuinely works right now; plug a real LLM into call_llm and you have a functioning app.

Making it production-grade

The structure stays identical — you just upgrade three pieces:

Here's the one meaningful code change — real embeddings (read-only sketch; needs pip install sentence-transformers):

from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("all-MiniLM-L6-v2")

def embed(text):
    return embedder.encode(text)   # now 'refund' ≈ 'money back'

With real embeddings, "money back" matches the refund chunk even though the words differ — that's the leap from keyword matching to meaning. Wrap ask() in a Streamlit UI and deploy it and you've got a shareable project.

Common pitfalls

• Chunks too big or too small: huge chunks dilute relevance; tiny ones lose context. Experiment.
• Retrieving too many chunks: more isn't better — it adds noise and cost. Start with k=3–5.
• Not citing sources: always return which chunks were used; it builds trust and aids debugging.
• Weak prompt: explicitly instruct "answer only from context, else say you don't know."

The big idea: a RAG app is just retrieve-then-prompt. Once you've built the skeleton, everything else is swapping in better parts.

Building my first RAG app end to end was the moment all the separate lessons — embeddings, vector search, prompting — suddenly connected into one thing that worked. Seeing it pull the exact right sentence out of my own notes and answer from it was far more satisfying than any tutorial I'd only read. My honest tip: build this skeleton first with the toy data, get it running, and only then swap in the real embeddings and LLM — debugging is ten times easier when each piece is simple.

Key takeaways

• A RAG app is six steps: documents → chunk → embed → store → retrieve → generate.
• The core is retrieve-then-prompt — find relevant chunks, then have the LLM answer using only them.
• You can build and run the retrieval today with just NumPy and cosine similarity.
• Go production by swapping in real embeddings (sentence-transformers), a vector DB (FAISS/Chroma), and a real LLM.
• Mind the details: chunk size, number of chunks, citing sources, and a strict prompt.

Continue the series: ← Lesson 30: Deploying AI Models  ·  Next: Lesson 32 — How to Evaluate LLMs and AI Systems →