RAG evaluation and testing
A RAG assistant answers from your documents, so it can fail in two places: it can fetch the wrong passages, or it can answer beyond what the passages say. We measure both separately, plus whether users can retrieve documents they shouldn’t see, so you know which part to fix.
What is RAG and why does it need its own tests?
RAG stands for retrieval-augmented generation. The system searches your documents, then asks a language model to answer using what it found. A wrong answer can come from the search or from the model, so end-to-end checks alone can’t tell you which one to fix.
What do you measure?
Retrieval: does the right passage appear in the results for each question. Groundedness: is every claim in the answer supported by the passages. Relevance: does the answer address the question. Citations: do they point to the right source. We also check what happens when the answer isn’t in the documents.
- Retrieval: the right passage is found
- Groundedness: claims are supported by the passages
- Relevance: the answer addresses the question
- Citations: sources point to the right place
- Out-of-scope questions: a clear "I don’t know"
Can users see documents they shouldn’t?
We test access control through the assistant. Using accounts with different permissions, we ask for content that only another role should see, and check that neither the answer nor the cited sources reveal it. Leaks here are a data protection problem, not only a quality problem.
What happens when we change the documents or settings?
Changes to chunking, embeddings, the search method or the document set can quietly make answers worse. We turn your golden set into a regression suite you run after each change, so a drop in retrieval or groundedness shows up before customers see it.
What does RAG evaluation cost?
Each project is quoted on its scope, so there’s no price list. The main cost drivers are how many documents and user roles there are, how often content changes, how many question types matter, and whether you want a regression suite. Request a quote and we reply by email.
Frequently asked questions
Do we need labelled data?
We build it with you. You approve question and answer pairs, and we mark which passages support each one. That golden set becomes yours.
Which RAG frameworks do you support?
We test behaviour, so the framework doesn’t matter. We need a way to send questions and, ideally, to see which passages were retrieved.
How often should we re-run the tests?
After any change to documents, prompts, models or retrieval settings, and on a schedule if your documents change often.
Ready to find out what is breaking?
Tell us what you’re shipping. We reply by email, and quote each project on its scope.