Evaluating AI Outputs

Chapter 6 cut cost with a semantic cache, but a cache hit is only a win if the cached answer was actually correct for the new question, not merely similar-sounding. More broadly, every change to a prompt, a model or a retrieval strategy raises the same question: did this make the system better or worse? Eyeballing a handful of outputs doesn't scale and doesn't catch regressions reliably. Evaluation is the discipline of scoring outputs against a reference answer systematically enough to actually trust the result.

7.1 Three Signals, Cheapest First

Production eval pipelines often use another LLM as a judge for nuanced grading, but a model call is exactly the expensive thing Chapters 1 and 6 were trying to avoid, so real pipelines typically run cheaper, deterministic checks first and reserve the model-as-judge step for cases that need it. This demo runs three of those cheaper signals: exact match (did the answer match character-for-character), a ROUGE-L overlap score (the longest sequence of words the candidate and reference share, in order; a standard, decades-old summarization and translation metric) and semantic similarity (the same embedding-based cosine similarity from Chapters 3 and 6, catching a correct answer that happens to be phrased differently).

Try the three samples above. The exact-match pair scores a perfect 1.0 on everything. The paraphrase scores low on exact match and ROUGE-L (neither cares about meaning, only shared words in order) while semantic similarity gives it partial credit for saying the same thing differently. The off-topic pair scores near zero across the board. No single signal is sufficient on its own, which is exactly why a real harness combines several.

Taken together, Chapter 1's token economics, Chapter 6's caching and this chapter's evaluation are the applied-engineering thread running underneath the rest of this tutorial: knowing what a request costs, cutting that cost without silently breaking correctness, and measuring correctness well enough to know the difference. Chapter 8 picks the story back up with the real-world deployments, at the millions-of-users scale, where getting exactly this right (or wrong) has real, measured consequences.