Overview
You upload documents, ask a question, and get an answer with numbered citations pointing at the chunks it came from. The backend is a FastAPI service over Qdrant. The frontend is a Next.js page that streams the answer token by token and shows the sources next to it.
I built it to learn retrieval properly: how the search strategies differ, how to measure them, and what it takes to keep a service like this alive on free hosting. Most of what I learned came from things that broke while the service looked fine.
What It Does
- Ingestion: PDF, DOCX, TXT and Markdown files are split into chunks of 500 characters with a 50 character overlap and stored in Qdrant Cloud with their metadata.
- Retrieval strategies: vector search, hybrid search (70% vector and 30% BM25, the default) and cross-encoder reranking. Reranking was switched off on the free deployment because it does not fit in memory; the hosted copy is offline now.
- Streamed answers: a server-sent events endpoint sends the citations first and then the answer tokens. There is also an OpenAI-compatible chat completions endpoint, so an existing OpenAI client can point at it by changing the base URL.
- Caching and limits: Redis caches answers and embeddings, with sliding window rate limiting per IP and endpoint. A repeated question measured 4.9 seconds cold and 0.16 seconds from the cache.
- Models: Ollama locally, Groq in the deployment, with a fallback between the two.
What I Measured
The evaluation harness (backend/tests/eval_retrieval.py) asks 20 questions against the deployed service (offline now, see the repo README) and checks whether the chunk that contains the answer was retrieved.
- hit@1: 85% for vector search and 90% for hybrid
- hit@3: 100% for all strategies
The corpus is 26 chunks, which is too small to separate the strategies, so hit@3 saturates and hit@1 is the number that discriminates. Reranking added nothing at this size, which says little about reranking in general. I read the result as: hybrid is a few points better than plain vector search on this corpus, and a larger corpus is needed to say more.
What Broke
A ranking bug that raised no error. The hybrid search converted a distance into a similarity with 1 / (1 + score). My Qdrant collection uses cosine similarity, where the score is already a similarity and higher is better. So the formula flipped the order: a 0.82 match scored 0.549 and a 0.12 match scored 0.893. The default search path was promoting the least relevant chunks. Nothing errored. It showed up as "I don't have that information" on questions that plain vector search answered correctly. I fixed it on 2026-08-03 by using the cosine score directly, then re-measured with the evaluation harness and got the numbers above. An older accuracy figure I had published predated the fix and was never re-checked, so I removed it.
A service that was down while every dashboard said it was fine. The Render instance has 512 MB of memory. The image shipped the CUDA build of torch, and the embedding model also loaded before the web server bound its port, so the process was killed during import and restarted over and over. From outside it looked like a very slow cold start. The fix went in layers: bake the model into the image, then drop torch entirely. Nothing in the request path needed it. It arrived through langchain_groq, then langchain_core, then an optional transformers import. Embeddings now run through fastembed on ONNX with the same all-MiniLM-L6-v2 weights, so the vectors are unchanged and the Qdrant collection needed no re-indexing. Import time went from about two minutes to 1.87 seconds, and peak memory with the model loaded went from 438 MiB to 259 MiB.
Failures that got cached. A failed generation was stored with the same one hour lifetime as a good answer, so one provider error became an hour of wrong answers. Failed generations are no longer cached.
A retired model. Groq retired the model the service named in seven places. The key stayed valid, so calls failed while the service looked healthy. The model name is now one setting with an environment override.
What I Learned
Retrieval quality matters more than the language model. A good model with bad retrieval answers confidently from the wrong chunks, and the inverted score showed me that I could not see that without measuring.
A health check that reports green is not a test. Several of these bugs were only visible when I ran the real question through the real route.
Free-tier limits shaped the design. The 512 MB limit decided the embedding stack, and the request quota decided the caching.
Tech Stack
Backend: FastAPI, LangChain, fastembed (ONNX), Ollama (local), Groq (deployed) Vector DB: Qdrant Cloud Search: BM25 (sparse), vector, cross-encoder reranking (off on the free deployment) Caching: Redis Frontend: Next.js 16, TypeScript, Tailwind CSS Deployment: Render and Vercel free tiers (suspended; run it locally from the README)