Refuse On Empty Context: The One Check That Stops RAG Hallucinations
A support bot confidently quotes a refund policy that doesn't exist. Nobody wrote a buggy prompt for that to happen — it happened because nothing checked whether the retrieval step actually found anything relevant before handing it to the model.
The problem: retrieval always returns something
A vector search doesn't fail the way a database lookup fails. Ask it for the nearest neighbors to a query, and it will return the nearest neighbors — even if the "nearest" ones are still conceptually miles away from the question. There's no built-in "I don't have that" response from a similarity search; it just returns its top-k results, ranked, no matter how weak the match actually is.
If your pipeline blindly stuffs those top-k chunks into the prompt and asks the model to answer, the model does what language models do with any context: it treats the provided text as authoritative and answers from it, even when that text has nothing to do with the actual question. The result looks confident and reads fluently — and is wrong.
The fix: turn distance into a similarity score, then threshold it
Vector search returns distances (how far apart two vectors are), not directly a 0-to-1 "how relevant is this" score. Smaller distance means a better match. A simple, widely used conversion:
similarity = 1 - distance(The exact formula depends on your distance metric — cosine distance maps to this cleanly; other metrics like Euclidean need their own normalization. Check what your vector store returns before using this formula verbatim.)
Once you have a similarity score, pick a threshold. Before generating an answer, check the top retrieved chunk's score against it:
- Clears the threshold → pass the retrieved context to the model and generate an answer from it, same as usual.
- Doesn't clear the threshold → skip generation from that context entirely and return a fixed response: "no relevant context found."
That's the entire safeguard. No fine-tuning, no extra model call — just a comparison and a branch.
results = vector_store.similarity_search_with_score(query, k=5)
top_score = 1 - results[0].distance # adapt to your store's distance metric
if top_score < SIMILARITY_THRESHOLD:
return "No relevant context found."
answer = generate(query, context=results)Why this matters more than prompt-level "don't make things up" instructions
Telling a model to "only answer from the provided context, and say you don't know if it's not there" helps, but it depends on the model correctly judging relevance from inside the same context window it's about to use to answer. A hard threshold check happens before the model ever sees the query — it's a deterministic gate, not a request for the model to police itself. Use both: the threshold as the hard gate, and prompt instructions as a second line of defense for cases that clear the bar but are only partially relevant.
Picking a threshold
- Run a batch of real queries — including ones you know have no good answer in your corpus — through retrieval and log the similarity scores.
- Look at where genuinely relevant results cluster versus where irrelevant ones cluster — the threshold sits between those two clusters.
- Re-check the threshold whenever you change embedding models or re-index your corpus; scores from different models aren't comparable to each other.
Power tricks
- Don't set the threshold so high that legitimate but loosely-worded questions get refused. Validate against real user queries, not just synthetic ones.
- A refusal on the top chunk isn't the same as a refusal on the whole corpus. If you retrieve k>1 chunks, decide whether you're thresholding the single best chunk or requiring some minimum number above the bar — the latter reduces single-lucky-match false positives.
- Log every refusal. A rising refusal rate is a signal — either your corpus has a real gap, or your indexing/chunking degraded and relevant content is scoring lower than it used to.
Resources
Building an AI feature? Yeda AI designs, audits, and ships production LLM systems.