An annotation queue is a workflow that sends model or agent outputs to human reviewers for labeling in a consistent, organized way. In AI evaluation, it turns scattered human judgments into consistent labels you can use to build datasets and check automated evaluators.
For example, suppose an agent’s factuality evaluator flags 400 production traces in a week, but your two reviewers can label only about 150. An annotation queue decides which 150 traces they see, in what order, with what rubric, and what happens when the two reviewers disagree.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why annotation queues matter for AI evaluation
Human review is the slowest and most expensive step in most evaluation pipelines. A queue makes sure that limited reviewer time produces labels you can trust and reuse.
Annotation queues are useful for:
- Validating automated evaluators: Compare human labels with evaluator scores to see where an LLM judge is wrong.
- Building evaluation datasets: Add reviewed cases, with their final labels, to a versioned dataset.
- Estimating production quality: Label a random sample to estimate how often the system fails.
- Fixing known failures: Focus reviewer time on flagged cases to understand what is going wrong.
A queue is a sampling and workflow decision before it is a tooling feature. Which cases get reviewed determines what the labels can tell you.
Arize’s overview of using annotations to collect human feedback walks through this flow for LLM applications.
How an annotation queue works
An annotation queue takes items from production traffic or an evaluation run. It shows each item to a reviewer with the context needed to judge it. It collects a structured label and stores that label against the original trace or example.
A working queue has five parts.
Selection
Selection decides which cases enter the queue. Common choices are a random sample, all failures flagged by an evaluator, low-confidence cases, high-traffic slices, or a mix.
Selection defines the population your labels describe. Labels from flagged cases describe flagged cases, not production as a whole.
Context packet
The context packet is what the reviewer sees. It can include the input, the output, retrieved documents, tool calls, and the trace around the step being judged.
Too little context produces guesses. Too much slows review and invites inconsistency.
Label schema
The label schema is the rubric the reviewer applies. It might be pass/fail, an error category, a numeric scale, free text, or a combination.
For more on schema design, see evaluation rubric and human labeling.
Assignment and ordering
Assignment decides who reviews what, in what order, and whether items get one reviewer or two. Priority ordering, such as newest failures first or highest-impact slice first, decides what gets labeled while reviewer time lasts.
Adjudication and export
Adjudication is how disagreements get resolved. Options include a second reviewer, a senior adjudicator, or a discussion step.
Export is how final labels move into datasets and evaluator comparisons. Without it, the review effort is spent once and lost.
Worked example: a failure-review queue
Return to the 400 evaluator-flagged traces and the capacity of about 150 reviews. A defensible queue design looks like this:
population: 400 evaluator-flagged traces
queue: - all 40 traces from the newest prompt version
- random 80 of the remaining failures
- 30 unflagged random traces (control sample)
double-review: 20% overlap between the two reviewers
labels: correct / wrong-fact / unsupported-claim / bad-citation
adjudication: disagreements resolved by a third reviewer each Friday
export: final labels appended to the regression dataset weekly
The three groups add up to 150 traces, which matches reviewer capacity. A 20% overlap means 30 of those traces get labeled by both reviewers.
The unflagged control sample is the part teams most often skip. Without it, you can estimate how often the evaluator’s flags are right. You cannot estimate how many failures it missed.
The double-reviewed overlap gives you an agreement rate. Suppose the reviewers agree on only 70% of the overlap. Then the schema or context packet needs work before you use the labels to judge an evaluator.
For more on that comparison, see measuring human-LLM judge alignment.
How to measure reviewer agreement
Percent agreement is the share of overlap items where both reviewers chose the same label. It is easy to read, but some agreement happens by chance.
For categorical labels, also report a chance-corrected measure such as Cohen’s kappa.
Formula:
κ = (p_o − p_e) / (1 − p_e)
Where:
- κ (kappa) is the chance-corrected agreement.
- p_o is the observed agreement, the share of items where both reviewers chose the same label.
- p_e is the agreement expected by chance, based on how often each reviewer uses each label.
The following code computes both numbers for a hypothetical set of 30 double-reviewed traces using the four labels from the example. It uses only the standard library.
from collections import Counter
# Hypothetical labels for the 30 double-reviewed traces.
pairs = (
[("correct", "correct")] * 9
+ [("wrong-fact", "wrong-fact")] * 6
+ [("unsupported-claim", "unsupported-claim")] * 4
+ [("bad-citation", "bad-citation")] * 2
+ [("unsupported-claim", "wrong-fact")] * 4
+ [("correct", "unsupported-claim")] * 3
+ [("bad-citation", "unsupported-claim")] * 2
)
n = len(pairs)
p_o = sum(a == b for a, b in pairs) / n
count_a = Counter(a for a, _ in pairs)
count_b = Counter(b for _, b in pairs)
p_e = sum(count_a[c] * count_b[c] for c in count_a) / n**2
kappa = (p_o - p_e) / (1 - p_e)
print(f"items: {n}")
print(f"observed agreement p_o = {p_o:.2f}")
print(f"chance agreement p_e = {p_e:.2f}")
print(f"Cohen's kappa = {kappa:.2f}")
Output:
items: 30
observed agreement p_o = 0.70
chance agreement p_e = 0.28
Cohen's kappa = 0.59
The reviewers agree on 70% of items, but chance alone would produce about 28% agreement. Kappa adjusts for that and comes out at 0.59.
The disagreements are also informative. Most of them involve “unsupported-claim,” which suggests the rubric does not separate it clearly from “wrong-fact.” Low agreement is a schema problem first and a reviewer problem second.
How to run and inspect a queue
Volume reviewed is not enough. Track whether the queue produces usable labels.
- Tracking the funnel: Count items selected, assigned, completed, adjudicated, and exported. A growing gap between selected and completed means the queue is a backlog, not a workflow.
- Measuring agreement: Report percent agreement and kappa on the double-reviewed overlap.
- Auditing final labels: Periodically re-review a slice of adjudicated labels against the rubric. This catches drift in how reviewers apply the rubric over time.
- Checking coverage by slice: Confirm the completed labels still represent the slices you care about. Priority ordering quietly skews coverage toward whatever was queued first.
- Closing the loop: Add exported labels to the evaluation dataset and compare them against automated evaluator scores.
For more on the wider practice, see human evaluation and dataset curation.
Limitations of annotation queues
A queue improves label quality, but it has known failure modes.
- Selection bias presented as ground truth: If only evaluator-flagged failures are reviewed, the labels describe flagged failures, not production quality. An evaluator tuned on them inherits the blind spots of the flagging evaluator.
- Reviewer fatigue: Long sessions degrade labels in ways that volume metrics hide.
- Anchoring: Showing an automated score next to the item pulls reviewers toward it.
- Rubric drift: Reviewers apply the rubric differently over weeks unless someone audits it.
- Unmanaged backlog: An oversized queue becomes a backlog that teams stop trusting. Set queue size by reviewer capacity and a sampling plan, not by how many traces a filter returns.
- Aging labels: A label applied under last quarter’s rubric or product policy may not describe what “correct” means now. Keep the schema version with every label.
Annotation queue vs. related workflows
Annotation queue vs. labeling tool: The labeling tool is the interface a reviewer clicks through. The queue is the selection, ordering, assignment, and adjudication system around it. Buying the tool does not decide your sampling strategy.
Annotation queue vs. evaluation dataset: The queue produces labels. The dataset is the durable, versioned collection those labels join. Each item should enter the dataset with its provenance: which trace, which schema version, and which adjudication outcome.
Annotation vs. feedback: Thumbs-up or thumbs-down signals and user reports are feedback. They are cheap, biased toward extreme reactions, and rarely based on a rubric. Queue annotation is deliberate review against a schema. Both have uses, but they are not interchangeable as ground truth.
Frequently asked questions
How do I choose which cases go into an annotation queue?
Start from the decision the labels will support. To estimate production failure rates, you need a random sample. To fix failures, you need flagged cases oversampled. To validate an evaluator, you need both flagged and unflagged cases. Most queues mix these groups deliberately and record which group each item came from.
How many reviewers should label each item?
Use a single reviewer for routine items, with a deliberate overlap that is double-reviewed to measure agreement. That overlap is often 10-20%. Items that decide a launch or change an evaluator’s threshold deserve full double review plus adjudication.
What agreement level is good enough?
There is no universal cutoff. Persistently low chance-corrected agreement means the labels cannot yet decide between model versions. Fix the rubric, examples, or context packet and re-measure before using the labels as ground truth.
Should reviewers see the automated evaluator’s score?
Decide deliberately. Showing it speeds review but anchors reviewers toward the evaluator, which inflates measured agreement. A common compromise is blind first-pass labeling, with the evaluator score revealed only during adjudication.
Why are my queue labels not improving my evaluator?
Usually the labels never leave the queue, or they come only from flagged cases. Export final labels into a versioned dataset, include an unflagged control sample, and compare the labels against evaluator scores on the same traces.
Putting annotation queues to work
Before you open a queue, write down the decision the labels will support, the sampling plan, the label schema version, and the overlap rate. Record which sampling group each item came from.
After each review cycle, report completion, percent agreement, and kappa. Export final labels with their trace ID and schema version.
Interface and API details for a specific product belong in its documentation rather than in this definition. If you run your queue in Arize Phoenix, see the Phoenix documentation.
