AI chatbot audit
A chatbot audit checks what your chatbot says against what you know is true. We build a golden set of questions from your own documents, run them repeatedly, check for leaks, unsafe replies and broken handoffs, and give you a ranked list of failures that your team can reproduce and fix.
What does a chatbot audit check?
Accuracy against your approved answers, whether it admits when it doesn’t know, whether it stays on topic, whether it leaks private or internal information, how it behaves over a long conversation, and whether it hands over to a person when it should. We also record response time and cost per answer.
- Accuracy against your approved answers
- Honest "I don’t know" when the answer isn’t available
- Staying on topic and inside its role
- No private or internal information in replies
- Consistency across a long conversation
- Handoff to a person when it should
How do you decide what a correct answer is?
You do. We draft questions and expected answers from your documents and support history, and you approve them. Approved answers become the golden dataset, the single source of truth. Where an answer can be worded many ways, we check the facts it must contain rather than the exact words.
What does the audit report look like?
Each finding has a title, a severity, the exact conversation that triggered it, how often it happened across repeated runs and the area to fix. Findings are grouped by impact so your team knows what to fix first. A QA engineer reviews every finding before it reaches you.
What do you need from us?
Access to a test or staging version of the chatbot, the documents it should answer from, a sample of real questions if you have them, and one person who can approve expected answers. We agree anything else during scoping, and we don’t need real customer data.
What does a chatbot audit cost?
Each project is quoted on its scope, so there’s no price list. The main cost drivers are the number of topics, languages and channels, how long conversations run, how many documents it answers from, and how much reporting you need. Request a quote and we reply by email.
Frequently asked questions
Can you audit a chatbot built on any model?
Yes. We test behaviour through the interface your customers use, so it doesn’t matter which model or vendor sits underneath.
Is this the same as a security test?
Partly. A chatbot audit includes basic leak and misuse checks. For a deeper attack-focused test, see prompt injection testing.
Can we repeat the audit after changes?
Yes. The golden dataset and test runs are yours, so they can be run again after each prompt, model or document change, by us or by your team.
Ready to find out what is breaking?
Tell us what you’re shipping. We reply by email, and quote each project on its scope.