How to Evaluate LLMs and AI Systems: Is Your AI Good?

You've built a RAG app (Lesson 31) and maybe fine-tuned a model (Lesson 29). Now the question that separates hobby projects from real systems: how good is it, actually? With classic ML you had clean numbers like accuracy (Lesson 11). With LLMs it's much trickier — there's often no single "correct" answer. Evaluation is the unglamorous skill that makes the difference between shipping something reliable and shipping something that sounds great and quietly fails. Let's make it concrete.

Why evaluating LLMs is hard

For a classification model, an answer is right or wrong — easy to score. But ask an LLM to "summarise this article" and there are many good answers and no exact target. Worse, LLMs are fluent: they produce confident, well-written text even when it's wrong, so "it sounds good" is a dangerous measure. You need methods that judge quality and correctness, not just plausibility.

The main ways to evaluate

LLM-as-a-judge: the modern workhorse

Since human eval doesn't scale and automatic metrics miss nuance, a popular middle path is to use a capable LLM to grade outputs. You give the judge the question, the answer, and clear criteria (or a reference), and ask for a score with reasoning.

Run a simple evaluation (runnable)

You don't need fancy tools to start. The foundation of all eval is a test set — questions paired with what a good answer must contain — and a scoring function. Here's a keyword-based check in pure Python:

# A tiny eval set: question -> keywords a good answer must include
eval_set = [
    {"q": "What is the refund window?", "must_have": ["30", "days"]},
    {"q": "When is the office open?",   "must_have": ["monday", "friday"]},
    {"q": "What do returns require?",   "must_have": ["receipt"]},
]

# Pretend these came from your AI system (from ask() in Lesson 31)
answers = [
    "Our refund window is 30 days from purchase.",
    "The office is open Monday to Friday.",
    "You mainly need the packaging.",          # missing 'receipt'!
]

def score(answer, keywords):
    a = answer.lower()
    return all(k in a for k in keywords)

passed = 0
for item, ans in zip(eval_set, answers):
    ok = score(ans, item["must_have"])
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'}: {item['q']}")

print(f"Score: {passed}/{len(eval_set)}")   # 2/3 — caught the bad answer

Crude? Yes. But it's automated, repeatable, and it caught a real failure. Run this every time you change a prompt or model and you'll instantly see if you made things better or worse — that's regression testing for AI.

Evaluating a RAG system specifically

RAG has two parts that can each fail, so evaluate both (this is why your ask() returned its sources in Lesson 31):

• Retrieval quality: did it fetch the right chunks? Measure with precision/recall on whether the needed passage was retrieved.
• Faithfulness (groundedness): is the answer actually supported by the retrieved text, or did the model add things? This is the big one for trust.
• Answer relevance: does it actually address the question asked?

A common setup uses LLM-as-a-judge for faithfulness: "Given these sources and this answer, is every claim supported? Yes/No, with reasons." Frameworks like RAGAS automate exactly these RAG metrics.

Testing for hallucinations

Since LLMs hallucinate, test for it deliberately: include questions whose answer isn't in your data and check the system says "I don't know" rather than inventing something. A model that confidently answers unanswerable questions is a red flag.

Offline vs. online evaluation

Two complementary stages: offline eval runs your test set before shipping (fast, safe iteration). Online eval measures real behaviour in production — A/B testing two versions, and tracking user signals (thumbs up/down, edits, retries). Offline tells you if it should work; online tells you if it does.

Build your own eval set

The single highest-leverage habit: create a "golden set" of 20–100 real questions with known-good answers for your task, and score every change against it. It doesn't need to be big to be useful — a small, honest test set beats vibes-based "seems better to me" every time.

The core discipline: if you can't measure it, you can't improve it. "It feels better" is not evaluation — a repeatable test set is.

The habit that changed the quality of everything I build was writing a small eval set before tweaking prompts, not after — the first time a "surely-better" prompt change actually dropped my score, I realised how often I'd been fooling myself with fluent-but-wrong outputs. Now I don't trust a change until the number moves. Start your golden set with just ten questions today; you'll catch regressions you'd otherwise ship.

Key takeaways

• LLMs are hard to evaluate because there's often no single right answer, and fluent text can still be wrong.
• Main methods: automatic metrics, human eval, LLM-as-a-judge, and benchmarks — each with trade-offs.
• For RAG, measure retrieval quality, faithfulness (groundedness), and answer relevance separately.
• Deliberately test for hallucinations, and use offline (test set) plus online (A/B) evaluation.
• Build a small golden eval set and score every change — that's regression testing for AI.

Continue the series: ← Lesson 31: Build a RAG App  ·  Next: Lesson 33 — LLM Security: Prompt Injection and How to Defend →