How to Evaluate AI Answers with a Small Test Set

An AI feature can produce a few impressive answers and still fail when real users depend on it. A demo usually contains friendly inputs and carefully chosen examples. Production traffic contains ambiguous requests, missing information, unusual formats, adversarial content and questions the system should not answer. A small, repeatable evaluation set helps you see whether a change actually improves the system.

This guide shows how to build an evaluation set for an AI assistant, define useful criteria, review failures and run the same checks after changing a prompt, model or retrieval pipeline.

1. Start by defining the task

Before collecting examples, write down what the system is supposed to do and what it must not do. “Answer questions well” is too vague to evaluate.

For a documentation assistant, a clearer task definition might be:

• Answer questions using the approved product documentation.
• Include a source reference for factual claims.
• Say when the documentation does not contain enough information.
• Do not invent product features, prices or policy exceptions.
• Do not reveal information the current user is not allowed to access.

These requirements become the evaluation criteria. They also help reviewers distinguish a stylistically pleasing answer from a correct one.

2. Build a representative test set

Start with 30–50 examples if that is enough to cover the main task patterns. A smaller, carefully designed set is more useful than hundreds of repetitive questions.

Include several categories:

1. Typical questions: common requests that the feature should handle.
2. Multi-part questions: requests that require combining information from more than one source.
3. Ambiguous questions: requests that need clarification or a stated assumption.
4. Missing-evidence questions: questions whose answers are not present in the supplied sources.
5. Out-of-scope questions: requests unrelated to the product or task.
6. Boundary cases: empty input, very long input, unusual formatting, malformed data or unsupported languages.
7. Safety and permission cases: requests that should be refused, restricted or escalated.

Do not build the entire test set from examples that the prompt author already knows how to answer. Include real support questions, anonymized user feedback and cases discovered during testing.

3. Write expected outcomes, not just ideal paragraphs

For each example, record what a correct response must contain. An exact reference answer is useful for some tasks, but open-ended generation often permits several valid phrasings.

A test record can include:

For a question about a product's refund window, the expected outcome might require the number of days and a citation to the refund policy. A response that sounds confident but cites the wrong policy should fail.

4. Evaluate several dimensions separately

A single score hides important failure modes. Track separate dimensions that reflect your task.

Factual correctness

Are the claims true according to the trusted source? Mark unsupported or contradictory statements. A fluent explanation is not evidence.

Completeness

Did the response address every part of the request? A response can be factually correct but omit a required condition or step.

Grounding and citations

Do the supplied sources support the claims beside them? Check that citations point to the correct document and section, not merely a document with a similar title.

Instruction following

Did the system use the required format, respect constraints and avoid prohibited actions? For structured output, validate the schema in code.

Refusal and uncertainty quality

When evidence is missing, does the assistant admit the limitation? Does it refuse unsafe or unauthorized requests without inventing a reason?

Operational quality

Measure latency, token usage, cost, timeouts and tool failures. A more accurate answer may still be unsuitable if it is far too slow or expensive for the product.

5. Keep retrieval and generation evaluations separate

For a RAG application, evaluate retrieval before judging the final answer. If the correct passage never reaches the model, a prompt change may not solve the root problem.

A retrieval test can record the relevant source passages and whether they appear in the first K results. An answer test can then check whether the model used those passages correctly.

This separation makes debugging much faster. It tells you whether to improve document parsing, chunking, search ranking, reranking, prompt instructions or the model itself.

6. Run a controlled experiment

A useful evaluation process changes one meaningful variable at a time.

1. Save the current prompt, model identifier and configuration.
2. Run the fixed test set and store the outputs.
3. Change one variable, such as the prompt or retrieval strategy.
4. Run exactly the same test set.
5. Compare scores and inspect every important regression.
6. Repeat the test before releasing the change.

Do not compare two configurations using different questions. The input distribution should remain stable enough for the comparison to be meaningful.

For stochastic generation, repeated runs may be useful. Record the settings and number of runs so the result can be reproduced.

7. Use automated checks and human review together

Automated checks are excellent for deterministic requirements: valid JSON, required fields, citation IDs that exist, forbidden strings, response length and expected status codes.

Human review is still important for nuanced correctness, clarity, tone and subtle unsupported claims. Model-based graders can help prioritize examples, but they can also share the same blind spots as the model being evaluated. Calibrate graders against human-reviewed examples and never treat an automated score as unquestionable truth.

8. Prioritize failures by impact

Not every error deserves the same response. A typo in a low-risk summary is different from exposing another user's data or inventing a legal requirement.

Classify failures by severity, frequency and affected workflow. Fix critical safety and privacy problems first. Then prioritize frequent factual errors and missing evidence. Keep a record of known limitations rather than quietly removing difficult examples from the test set.

9. A simple evaluation report

A useful report should show more than one overall percentage. Include:

• Number of test cases and date of the test.
• Model and configuration used.
• Correctness and completeness results.
• Grounding and citation results.
• Refusal and escalation results.
• Latency and cost summaries.
• Regressions compared with the previous version.
• Representative failures and planned fixes.

Keep the original outputs. They help explain why a score changed and make future debugging possible.

Common mistakes

Frequently asked questions

How many examples do I need?

Start with enough cases to cover the main task categories. Thirty to fifty is a practical first set for a narrow feature, but high-risk or complex applications need broader coverage and ongoing sampling from real use.

Should I use exact expected answers?

Use exact answers when output must be deterministic, such as a classification label or structured field. For open-ended responses, define required facts, source evidence and unacceptable claims instead.

Can an LLM evaluate another LLM?

It can assist with grading, but it should not be the only authority. Compare grader decisions with human-reviewed examples and investigate disagreement.

Final checklist

• Define the task and boundaries before writing tests.
• Include normal, edge, ambiguous and unsupported cases.
• Record required facts, sources and forbidden claims.
• Score correctness, completeness, grounding and safety separately.
• Evaluate retrieval independently from generation.
• Change one variable at a time.
• Combine automated validation with human review.
• Preserve outputs and track regressions over time.

A small evaluation set is not a one-time launch checklist. It is a living regression suite for an AI feature. Keep it representative, review failures honestly and rerun it whenever the system changes.