Dataset curation is the practice of selecting, cleaning, labeling, organizing, and maintaining the examples used for evaluation. It covers removing duplicates, balancing coverage, adding hard cases, preserving metadata, and deciding which production traces should become regression tests.
Curation is not busywork. The dataset defines what the eval suite can see, so it sets a ceiling on what any evaluator can tell you. A small curated set of real failures is routinely more useful than a large synthetic set with shallow coverage, and the gap between those outcomes is a series of small judgment calls about what to include and when to stop changing the set.
This is the practice, not the plumbing. Where examples are stored, how versions are implemented, and how a row points back to its originating trace are properties of the datasets layer. Curation is what you decide to put in it.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Curate for coverage of behaviors, not for volume. A few hundred examples spanning distinct behaviors beat thousands of near-duplicates measuring one thing repeatedly.
- A set built only from failures cannot detect a regression on anything that currently works, and it exaggerates every fix. Keep a representative base set and a separate failure library.
- Near-duplicates quietly reweight your metric. Twenty paraphrases of one request give that request twenty votes in the average.
- Every change to a dataset creates a new version, because scores from a changed set are not comparable to scores from the old one.
- Cases that have passed for months have stopped testing anything. Auditing the always-green portion is curation work, not cleanup for later.
How to decide what goes in
Start from the behaviors you need to hold, not from the rows you happen to have.
Stratify against real traffic. List the dimensions that actually change behavior: intent or task type, customer segment, language, input length, whether retrieval is involved, which tools are in play. Then give each cell enough examples to read on its own. If half your traffic is one intent and your set has three examples of it, the aggregate score is describing something other than your product.
Oversample consequence, not frequency. Rare inputs with expensive outcomes deserve more room than their traffic share: billing edge cases, phrasing your policy forbids answering, injection attempts, requests that touch another tenant’s data. You are deliberately distorting the distribution, which is fine as long as those slices are scored separately rather than blended into one number.
Keep the boring cases. The mundane happy path is what regressions break first and the least interesting thing to add, so it is usually missing. A set made entirely of interesting cases will not notice when the simple ones start failing.
Write the cases traffic will never hand you. Empty inputs, truncated messages, mixed languages, contradictory instructions. These rarely appear often enough to be caught by sampling, so hand-writing them is a legitimate start, and getting that set into a platform is usually a matter of creating a dataset from a CSV before you grow it from production.
Why curating only from failures overfits
The failure-driven habit is right in spirit. Real failures are the sharpest examples you will get, and adding each incident to the set the week it happens is the best curation reflex there is. The problem is a set composed of nothing else.
Three things go wrong. You lose sight of collateral damage: if every case is something the system got wrong, nothing is watching the behaviors it got right, so a prompt change that fixes ten failures and breaks a common path scores as a clean win. The score also stops meaning anything in absolute terms, because a failure-only set is adversarial by construction, so 60% is an artifact of how you built it rather than a quality level. And iteration converges on the specific cases in front of you, which is fitting to the set rather than improving generalization.
The workable structure is two sets with different jobs. A representative set that mirrors production tells you where you stand. A failure library, grouped by failure category, tells you whether known problems came back. Report them separately, because averaging them produces a number that answers neither question.
Cleaning, labeling, and metadata
Deduplicate on meaning, not string equality. Production traffic is repetitive, so naive sampling pulls many variants of the same request. Cluster by embedding similarity or group by intent, keep a handful of variants where phrasing genuinely matters, drop the rest.
Decide what ground truth means per case type. Some cases have a reference answer, many do not, and forcing one is how a set ends up penalizing correct outputs that differ in wording. For open-ended tasks the durable artifact is the criteria a judge applies, and a curated set of labeled examples is what you calibrate that judge against. That calibration is the substance of building a custom evaluator against a benchmark dataset: without human labels on real examples, you have no way to know whether the judge agrees with you.
Record metadata when you promote, never later. Source, date, segment, language, difficulty tier, and failure category cost seconds at write time and are painful to reconstruct. They are what turns a single score into a diagnosis.
Public and open sets have a place. They help bootstrap a domain where you have no traffic and let you compare against work outside your team, as with an open dataset built for hallucination detection in retrieval systems. Treat them as a supplement: they measure general behavior rather than your application, and widely circulated sets may already have been absorbed into model training.
Curation is a loop, and versions make it safe
A dataset that never changes goes stale. A dataset that changes without versions destroys your history. Both come from treating the set as a static asset rather than a maintained one.
The loop that keeps a set honest: add production incidents as they happen, refresh a rotating slice from recent traffic on a cadence tied to how fast the product moves, review reference answers whenever the policy or corpus behind them changes, retire cases that no longer test anything, and stamp a new version every time you do any of it. The version is what makes the rest reversible. Without it, every score in your history refers to a dataset that no longer exists.
FAQ
How many examples should a curated dataset have?
Enough that each slice you report has a stable number, which usually means starting in the low hundreds and growing from real failures. The useful test is repeatability: if a metric swings several points between runs with nothing changed, the slice driving that swing needs more examples, and adding rows elsewhere will not help.
Should I curate examples only from failures?
Add every failure, but do not build the whole set that way. A failure-only set cannot detect regressions on working behavior, produces a score with no absolute meaning, and encourages tuning against the specific cases in it. Keep a representative base set alongside the failure library and read them separately.
How do I know when a dataset needs refreshing?
Compare it to a recent traffic sample on the dimensions you tagged. New intents, a new language, longer inputs, or a large customer with different vocabulary show up as divergence between set and traffic. A rising pass rate with no matching drop in user complaints is the other signal, and it usually means the set has aged out of relevance.
Who should own curation?
Whoever owns the eval suite, with named owners for reference answers that depend on policy or documentation. Curation fails most often through diffusion of responsibility: the set belongs to everyone, so the stale case nobody wrote stays in it for a year.
