The Symptom
My Enterprise RAG project answers questions over uploaded documents. Its default retrieval is hybrid search, which blends vector similarity with BM25 keyword matching. For a while it kept saying "I don't have that information" to questions that the plain vector search answered correctly when I tried it.
There were no exceptions and nothing odd in the logs. The service did what it was told. What it was told was wrong.
The Cause
The hybrid search code converted a distance into a similarity with 1 / (1 + score). That formula is right for a distance, where lower is better. My Qdrant collection uses cosine similarity, and for cosine, Qdrant already returns a similarity where higher is better.
So the formula flipped the order. A chunk with a similarity of 0.82 scored 0.549, and a chunk with a similarity of 0.12 scored 0.893. The best match went to the bottom and the worst one went to the top, and the language model was then asked to answer from the worst chunks.
I fixed it by using the cosine score directly, clamped between 0 and 1.
Why Nothing Caught It
Every layer did its own job correctly. The search returned results, the model answered from whatever it was given, and the API returned a response. A health check would have been green. Reading a few answers by hand would not have been enough either, because a wrong answer from the wrong chunk sounds like a normal answer.
The only way to see it was to measure retrieval separately from generation. I wrote an evaluation harness that asks a set of questions and checks whether the chunk containing the answer was retrieved at all. Run against the deployed service on 2026-08-03 after the fix, it gave a hit@1 of 85% for vector search and 90% for hybrid, and a hit@3 of 100%.
What the Numbers Do and Do Not Say
The test corpus is 26 chunks and there are 20 questions. That is small enough that hit@3 is saturated: every strategy finds the right chunk within the top three. So the harness cannot separate the strategies well, and hit@1 is the only number with any resolution. Hybrid ahead of vector by five points is the most I can claim.
The reranker showed no gain at this size, which is what I would expect when there is barely anything to reorder. It does not show that reranking is useless.
What I Changed Afterwards
I removed the accuracy figures I had published earlier. They were measured before the fix, on a different corpus, and I had never re-run them through the corrected code. A number that came from a bugged scorer is not worth keeping, even if it looks good on a page.
A ranking bug makes a product quietly worse instead of loudly broken, so it needs a test that looks at the ranking itself.
The project is on GitHub, and the harness is backend/tests/eval_retrieval.py.