Semantic Caching

Chapter 1 put a real number on every request to a hosted model. The cheapest possible request is one that never has to be made at all, and a cache is the standard way to avoid making it. A plain cache only helps when a new request's text matches a previous one exactly, character for character. Real user traffic rarely cooperates: two people asking "What is the Model Context Protocol?" and "Can you explain MCP to me?" are asking the same question, but no exact-match cache would ever notice.

6.1 Matching by Meaning, Not by Characters

A semantic cache fixes this by keying on meaning instead of exact text: each incoming query is embedded (the same technique from Chapter 3) and compared by cosine similarity against every embedding already sitting in the cache. If the closest match clears a similarity threshold, it's treated as the same question and the cached result is reused; the model is never called at all. If nothing clears the threshold, the query is a genuine miss, gets embedded and is added to the cache for future queries to match against.

Try the sample query stream above: the second query is a paraphrase of the first, but their real embeddings typically land only around 0.4–0.5 cosine similarity, well short of the 0.85 default, so it registers as a miss. That's a genuine, measured property of short, differently-worded questions like these, not a shortcut this demo is taking: the embeddings here are real, but two people phrasing the same question differently still don't land as close together in vector space as intuition might suggest. The fourth query, an exact repeat of the first, hits immediately at similarity 1.0. Lower the threshold to somewhere around 0.4–0.5 and rerun to see the paraphrase get caught too.

6.2 The Threshold Is a Real Trade-off

Set the threshold too low, and unrelated questions start getting the same cached answer: a correctness problem, not just an efficiency one. Set it too high, and near-duplicate questions stop matching, and the cache barely saves anything. There's no universally correct value; production systems tune it against real traffic and measured accuracy loss, which is precisely the kind of measurement Chapter 7 covers next.