AI testing and QA glossary

An AI hallucination is when a model states something false or unsupported as if it were true. It can invent facts, sources, quotes or policies, and it usually sounds confident. You catch hallucinations by comparing answers with approved facts and source documents, not by how convincing the answer sounds.

See also: Chatbot audit

AI regression testing means rerunning the same evaluation after anything changes, such as the prompt, the model, the documents or the settings. It shows whether quality dropped. Because AI answers vary, each test runs several times and you compare pass rates, not a single pass or fail.

See also: AI testing

CI/CD stands for continuous integration and continuous delivery. Every code change is built and checked automatically by a pipeline, and the checks can include your tests. A change that breaks a test can be stopped before it reaches customers. Playwright tests usually run in CI so every pull request gets checked.

See also: CI/CD integration

End-to-end testing checks a whole user journey the way a real person would do it, from the screen in the browser through the backend and back. Examples are signing up, logging in or checking out. It catches problems that only appear when the parts work together, but it runs slower than smaller tests.

See also: Playwright

A flaky test passes and fails on the same code without any change to the product. Common causes are fixed waits, shared test data and unstable selectors. Flaky tests erode trust, because people start ignoring red builds. We fix the cause instead of hiding it behind automatic retries.

See also: Test automation services

A golden dataset is a set of questions with expected answers that you have approved. It's the source of truth for what correct means. We build it from your documents and support history, and when the AI and the golden dataset disagree, the golden dataset wins. It stays yours.

See also: Methodology

Grounding means an AI answer is tied to source material, so each claim can be traced to a document. A grounded answer says only what the sources support. An ungrounded one adds details the documents never mention. Testing grounding means checking every claim in an answer against the passages the system retrieved.

See also: RAG evaluation

A jailbreak is a prompt designed to make a model ignore its safety rules, such as by role-play or tricky wording. It's related to prompt injection but not the same. A jailbreak targets the model's own restrictions, while prompt injection targets the instructions of the app built on top of it.

See also: Prompt injection testing

LLM-as-a-judge means using one language model to score another model's output against a written rubric. It's useful for fuzzy qualities like tone or clarity, but it can be wrong or biased. So it should quote its evidence, use a different model from the one under test, and be checked against human labels.

See also: Methodology

LLM evaluation, often called evals, is the practice of measuring how well a language-model product does its job. An eval combines test questions, checks written in code, human review and sometimes an AI judge. Good evals are repeatable, so you can tell whether a change made the product better or worse.

See also: AI testing

Playwright is an open-source tool for automating and testing web browsers, built by Microsoft. One API drives Chromium, Firefox and WebKit. It waits for elements to be ready before acting, runs tests in parallel and records traces you can replay. You can write tests in JavaScript, TypeScript, Python, Java or .NET.

See also: Playwright

Prompt injection is an attack where crafted text makes an AI follow the attacker's instructions instead of yours. The text can come straight from a user or hide inside content the AI reads, like a document, email or web page. It's listed first in the OWASP Top 10 for LLM applications.

See also: Prompt injection testing

RAG, short for retrieval-augmented generation, is a way to make an AI answer from your own documents. The system first searches your content for relevant passages, then asks a language model to write an answer using only what it found. Answers can fail in the search or in the writing.

See also: RAG evaluation

Red teaming means deliberately attacking your own system from an adversary's point of view to find weaknesses before real attackers do. For AI products that includes trying to extract private data, bypass rules and misuse tools. A good red team records every attempt, so findings can be reproduced and retested after fixes.

See also: Prompt injection testing

Regression testing checks that features that used to work still work after a change. Every new release risks breaking something old, so teams rerun a set of tests each time. Automating those tests makes the checks fast and repeatable, and it frees people to explore new features instead of repeating the same clicks.

See also: Test automation services

A Selenium to Playwright migration moves an existing test suite to Playwright without losing protection along the way. It works best in stages: group tests by risk, rewrite one batch, run old and new side by side, and retire the original only after the new test fails when the app is deliberately broken.

See also: Selenium and Cypress migration

Test automation means using code to run tests instead of checking by hand. A script opens the app, performs the steps and compares the result with what's expected, the same way every time. It's best for repetitive checks on your most important journeys. Exploratory and design questions still need a person.

See also: Test automation services

Test coverage describes how much of your app's behaviour or code your tests exercise. A high number doesn't mean good tests, because a test can run code without checking anything useful. We prefer to cover the riskiest user journeys first, then widen coverage where bugs actually cost you time or customers.

See also: Test automation services