Retrieval-Augmented Generation (RAG)

A language model's knowledge is frozen at whatever point its training data was collected, and it has no built-in way to consult anything outside that data. This creates two related problems: it cannot answer accurately about information that didn't exist at training time, or that was never public in the first place, and when asked about something it doesn't actually know, it tends to produce a confident, fluent-sounding answer anyway, a failure mode usually called hallucination, since the model isn't lying so much as filling a gap the only way it knows how.

Retrieval-Augmented Generation, or RAG, addresses this by splitting the task into two stages instead of relying on the model alone. A retrieval stage first searches a separate collection of documents for the passages most relevant to the question being asked. A generation stage then hands those retrieved passages to the language model as part of its instructions, alongside the original question, and asks it to answer using only that supplied material. The model is no longer answering purely from memory; it's reading source material at the moment it responds, the same way a person might check a reference before answering rather than guessing.

The retrieval stage itself works through embeddings. An embedding is a list of numbers that represents the meaning of a piece of text as a position in a high-dimensional space, produced by a model trained so that texts with similar meaning end up positioned close together, and unrelated texts end up far apart. Every passage in the document collection is converted into an embedding once, ahead of time, and stored. When a question arrives, it's converted into an embedding the same way, and the system finds the passages whose embeddings sit closest to the question's, typically using cosine similarity, a measure of how closely two of these positions point in the same direction. The closest matches are the passages judged most relevant.

The demo above only has a few dozen passages to search, few enough that comparing a question's embedding directly against every stored one is instant either way. Real systems often store millions or billions of embeddings, where checking every single one for every question stops being practical. Production vector databases handle this with an approximate index, most commonly one called HNSW (Hierarchical Navigable Small World), which arranges embeddings into a graph structured so a search only has to follow a handful of connections to land near the true closest matches, rather than visiting every stored embedding directly. That trades a small, usually unnoticeable amount of accuracy for a very large speed gain at scale, exactly the trade-off Chapter 8 shows a real company making, and exactly the index this tutorial's own vector database actually runs, covered in more detail in Chapter 9.

Finally, the model is instructed to answer strictly from the retrieved passages and to cite which passage supports each part of its answer, so instead of taking the response on faith, you can check exactly where each claim came from. That's what you should have just seen above: the answer itself, and the numbered source passages underneath it that were retrieved to produce that answer.