Labeling is the process of attaching judgments to examples. Human labeling is the version where people do it: reviewers mark correctness, relevance, safety, preference, task success, or any other property you need recorded against a specific record. The product of the work is a dataset of examples with agreed-upon labels, which is what almost everything downstream depends on.
That dependency is why labeling deserves more engineering attention than it gets. An LLM judge is validated against human labels. A release gate is calibrated against them. A fine-tune learns from them. When the labels are inconsistent, none of those things fails loudly. They keep producing numbers, and the numbers are wrong by however much the labels disagree with each other.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Human labeling produces the ground truth other evaluation depends on, so its error rate becomes the error floor for every measurement built on it.
- The written guideline is the real deliverable. Most label noise traces back to an instruction that did not cover the case in front of the reviewer.
- Overlap a portion of the work between reviewers and measure inter-annotator agreement. Without it you have labels and no idea whether they mean anything.
- Disagreement is a signal about the guideline, not a verdict on the reviewer. Adjudicate the conflict, then write the resolution back into the instructions.
- Sample deliberately. Uniformly sampled labels spend most of the budget on cases the system already handles.
What gets labeled
In an LLM or agent application the labelable unit is rarely just a final response. Depending on the question, reviewers work on:
- Spans. A single retrieval, tool, or model call. Labeling a retrieval span for relevance is how you find out whether a bad answer was a retrieval problem or a generation problem.
- Traces. One full request, including every step the agent took, labeled for whether the right tool was chosen and whether the sequence made sense.
- Sessions. A multi-turn conversation, labeled for whether the user’s goal was ever met.
- Pairs. Two candidate outputs for the same input, labeled by preference.
Label shape matters as much as label target. Binary is easiest to aggregate and alert on. A short categorical set works when the levels are genuinely distinct. A free-text note field belongs on all of them, because the reviewer’s one-line reason is what makes a label reviewable later.
The guideline is the deliverable
A labeling guideline is a specification, and like any specification its bugs show up as inconsistent output. A usable one states:
- The question, exactly as the reviewer should ask it. “Is every factual claim in this answer supported by the retrieved documents?” not “is this answer good?”
- The label set, with a definition per value written as an observable condition.
- What evidence is in scope. May the reviewer use outside knowledge, or only the retrieved context? Does a harmless but unhelpful refusal count as a failure? Unstated scope is where two careful people diverge.
- Worked examples, including the hard ones, taken from real traffic.
- What to do when unsure. An explicit abstain value beats a coin flip recorded as data.
Version the guideline. When a rule changes mid-project, labels produced before and after are two different measurements, and pooling them silently is how a dataset ends up contradicting itself.
Agreement and adjudication
Assign a slice of items to two or more reviewers independently, then measure how often they land on the same label. Raw percent agreement is easiest to read but flatters you on skewed tasks, since two reviewers who both label almost everything pass will agree often by accident. Chance-corrected measures such as Cohen’s kappa for two raters, or Krippendorff’s alpha when the number of raters varies, are the standard correction.
What you do with a low number matters more than the statistic. Low agreement is almost never a staffing problem. Pull the items reviewers split on and read them: they usually cluster into two or three recurring situations the guideline never addressed. Fix the definition, add the disputed items as worked examples, and re-run the overlap. Treating disagreement between reviewers as data worth keeping rather than noise to average away is what turns a labeling program into something that improves.
Adjudication handles the items that stay contested. A senior reviewer or the guideline owner makes the call and records the reasoning, and that adjudicated label becomes the gold version. Keep those items separate. They are the hardest and most informative part of any evaluation set.
Where the labels get used
The most common consumer is judge validation. You label a sample, run the automated judge over the same sample, and compare. That comparison is what makes an automated eval reportable, and measuring how closely a judge matches human labels is the difference between an eval you can defend and one you hope is working.
Labels also feed threshold setting, regression datasets, fine-tuning data, and prompt iteration. Most of that flows through a queue rather than a spreadsheet, and collecting annotations against the application’s own traces keeps each label attached to the context that produced it, which is what makes it reusable six months later.
Failure modes
One annotator per item, forever. No overlap means no agreement number, which means no evidence the labels are reliable.
The abstain bucket as a dumping ground. If a third of items land in “unclear,” the guideline is underspecified, not the data.
Gold set contamination. If your ground truth came from the model you are evaluating, the eval cannot see that model’s blind spots. Keep one human-labeled set no model touched.
Uniform sampling. Random samples over production traffic mostly reproduce the common case. Stratify by score, by flagged status, by customer tier, by newly deployed prompt version.
Throughput as the metric. Items per hour is easy to track and rewards guessing. Agreement rate and adjudication rate tell you whether the work is any good.
FAQ
What is the difference between labeling and annotation?
In practice they are interchangeable. Where teams do draw a line, annotation means attaching a judgment to a specific record, and labeling means the broader program of producing a dataset.
What counts as good inter-annotator agreement?
It depends on the task, and any universal threshold you see quoted deserves suspicion. A binary policy check with crisp rules should reach agreement in the high nineties; a subjective helpfulness comparison will not, no matter how good the guideline. The useful framing is that whatever agreement your reviewers reach is the ceiling on how well any automated eval can be expected to match them. Report it alongside the eval score.
How many labels do I need?
Fewer than most people assume for calibration, more than most budget for coverage. A few hundred well-adjudicated items is often enough to validate a judge on one dimension. Confirming behavior across many segments, languages, or intents needs enough items per slice to say anything about that slice, which is where the number grows.
Can a model do the labeling instead?
For some purposes, yes, and that is what weak supervision and LLM-as-a-judge labeling are. Both trade accuracy for volume, and both inherit the labeling model’s blind spots. The workable arrangement is programmatic labels for breadth, a smaller human-labeled and adjudicated set as the reference, and a measured agreement number between them.
What is the difference between human labeling and human evaluation?
Labeling produces the annotations and ground truth that everything else depends on, including the guidelines, agreement checks, and adjudication that make those annotations trustworthy. Human evaluation is people judging output quality, usually to produce a score or a preference. Human evaluation is typically a consumer of labeling work.
