An evaluation dataset is a collection of examples used to test an AI system. Each example may include inputs, expected outputs, retrieved context, labels, metadata, traces, or scoring criteria, depending on what the eval needs to see.
It is the artifact that makes an evaluation reproducible. Criteria and scoring mechanisms can stay identical between two runs, but if the examples change, the scores are not comparable and the comparison you were trying to make falls apart. A versioned dataset is what turns “the new prompt feels better” into a number you can defend.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- An evaluation dataset is held out from training and used only for measurement. Any overlap with training data inflates your results.
- The best examples come from production: real failures, edge cases, high-value tasks, and policy-sensitive inputs, not invented queries.
- Datasets need versions. Comparing a score from today against one from last month is only valid if you know which version of the data produced each.
- Coverage matters more than size. A few hundred examples spanning your real input distribution beat ten thousand near-duplicates.
- Every evaluation dataset goes stale as your product, users, and model change. Treat refreshing it as recurring work, not a one-time setup.
What goes in an example
The minimum is an input. Everything else depends on what you are scoring.
For a factual question-answering system you often want an input, a reference answer, and the documents that should have been retrieved. For a summarization or drafting task there is usually no reference answer, so the example carries the input plus the criteria a judge will apply. For an agent, the useful unit is often the whole trace: the user request, the sequence of tool calls with their arguments and returns, and the final response, since the path is part of what you are grading.
Metadata is what makes the dataset useful later. Tag each example with its source, customer segment, language, difficulty, and failure category. Those tags are what let you slice results afterward, and adding them at collection time costs almost nothing while adding them retroactively is miserable.
Where the examples come from
Production traces. This is the primary source and the one that produces the sharpest datasets. Sample real traffic, review the outputs, and promote the interesting ones into the dataset. Curating examples directly from the traces your application already produces means every case is one a real user actually sent, with the retrieved context and tool calls attached.
Handwritten edge cases. Adversarial inputs, prompt injection attempts, ambiguous requests, empty or malformed inputs, and the specific scenarios your policy team cares about. These rarely appear often enough in sampled traffic to show up on their own, and they are usually the ones with the worst consequences.
Synthetic generation. A model can expand a small seed set into many variations, useful for covering phrasings and for testing a feature that has no traffic yet. The caveat is real: synthetic examples reflect the generating model’s idea of what users do, which is smoother and more grammatical than what users actually do. Mix them in, do not build on them alone.
Public benchmarks. Standard datasets are useful for capability comparison and for bootstrapping a domain you have no data in. There is also open-source work on shared datasets for specific failure modes, including an open dataset for hallucination detection in retrieval systems. Just remember that a public set measures general behavior, not your application, and that popular ones may have leaked into model training.
Evaluation data is not training data
The distinction is old and still gets broken constantly. Training data teaches the system. Evaluation data measures it, and it only measures honestly if the system has never seen it. Once an example influences a prompt, a fine-tune, or a retrieval index, its score stops being a prediction of behavior on new inputs.
In LLM applications the leak is usually subtle rather than explicit. Nobody fine-tunes on the eval set. What happens instead is that an engineer stares at fifty failing eval examples, edits the prompt until they pass, and reports the improved score. That is fitting the prompt to the test. The defense is a holdout split you do not look at, plus enough fresh examples arriving from production that the set cannot be memorized.
The same logic applies to public benchmark datasets and their limits, where contamination from web-scale pretraining makes some scores hard to interpret.
Keeping it representative
A dataset is representative when the mix of examples resembles the mix of real traffic, with deliberate over-sampling of rare cases that matter. That balance decays. Users discover new ways to use the product, you ship a feature that changes the input distribution, a large customer onboards with a different vocabulary, and the set that described your traffic in March describes something else by September.
Three habits keep it current. Compare your dataset’s distribution against a recent production sample on the dimensions you tagged. Add every production incident as an example the same week it happens, so the set accumulates real regressions. And review the examples that always pass, because a case nobody has failed in six months is no longer testing anything.
Size is usually less of a problem than people expect. A few hundred well-chosen examples covering distinct behaviors gives a usable signal. Ten thousand paraphrases of the same three questions measures one narrow thing precisely.
FAQ
How large should an evaluation dataset be?
Large enough that the metric is stable between runs and each slice you care about has enough examples to mean something. In practice, teams start in the low hundreds and grow from failures. If a metric swings several points when nothing changed, you need more examples in that slice, not more examples overall.
What is the difference between an evaluation dataset and a test set?
They are the same idea, with the term shifting by context. Classical machine learning splits data into train, validation, and test. LLM application work usually has no training step, so “evaluation dataset” is the common name for the curated examples used to measure behavior. The holdout principle carries over unchanged.
How do I build an evaluation dataset for tool use or agents?
Store the full trajectory, not just the final answer. Each example should include the user request, the tools that were available, the sequence of calls with arguments, and what each call returned. That lets you score whether the agent picked the right tool, passed valid arguments, and recovered from failures, which answer-only examples cannot distinguish.
Can synthetic data replace real user data?
Not on its own. Synthetic examples are good for coverage, for pre-launch testing, and for expanding a thin category. They systematically miss the typos, truncated questions, mixed languages, and strange framing that real users produce, and those are frequently where systems break.
How often should the dataset be refreshed?
Continuously in small amounts rather than occasionally in large ones. Add production failures as they occur, and audit the distribution against live traffic on a regular cadence, monthly for a fast-moving product. Version each change so old scores remain interpretable.
