An evaluation rubric is a structured set of criteria used to score AI outputs or agent behavior. It tells a human reviewer, an LLM judge, or a scoring function what good and bad look like, in enough detail that two different reviewers reach the same verdict on the same output.
Rubrics are where subjective quality becomes operational. Everyone on a team agrees the assistant should be “helpful” and nobody agrees on which of two specific answers is more helpful. The rubric is the artifact that settles that argument once, in writing, before anyone starts scoring. Skip it and you do not avoid the subjectivity, you just move it into each individual scoring decision where it becomes invisible.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- A rubric turns a quality goal into explicit levels with definitions, so two reviewers scoring the same output land in the same place.
- Vague criteria are the main source of noisy evals. “Rate helpfulness 1 to 10” produces disagreement that looks like signal.
- Every level needs a worked example. Definitions drift in interpretation; examples anchor them.
- Test the rubric before trusting it: have two people score the same sample and measure how often they agree.
- The rubric is the criteria. The scoring function is the mechanism that applies them. Keeping them separate lets you change one without invalidating the other.
What a rubric contains
One dimension per rubric. Faithfulness and tone are separate rubrics, not one combined score. A reviewer asked to weigh an ungrounded but pleasant answer against an accurate but blunt one will resolve that tradeoff differently every time.
A small scale with named levels. Binary works well for compliance-style checks: grounded or not grounded. Three or four levels work for graded quality. Ten-point scales are almost always a mistake, because nobody can articulate the difference between a 6 and a 7, and both humans and LLM judges compress their scores toward the middle anyway.
A definition for each level, written as an observable condition rather than an adjective. “Every factual claim appears in the retrieved context” is checkable. “Mostly accurate” is not.
Evidence rules. State what the evaluator may look at. Should a groundedness score consider the model’s own prior turns, or only the retrieved documents? Should a helpful answer that ignores the requested format lose points on helpfulness, or is format a separate rubric? Unstated scope is where reviewers silently diverge.
Worked examples for each level, taken from real outputs. This is the part teams skip and the part that does the most work.
Vague criteria are the failure mode
Consider the difference between two instructions. The first: “Score the answer’s helpfulness from 1 to 10.” The second: a four-level rubric where level 0 means the answer does not address the question asked, level 1 means it addresses the question but omits information the user needs to act, level 2 means it fully answers the question asked, and level 3 means it fully answers the question and correctly surfaces a constraint the user did not ask about but needed.
The first produces a distribution centered on 7 with disagreement everywhere. The second produces disagreement only on genuinely ambiguous cases, and when reviewers disagree you can read the level definitions and see which clause was interpreted differently. That is the property you want: a rubric should localize disagreement, not hide it.
The same discipline is what makes an LLM judge usable. A judge given a leveled rubric with examples returns far more consistent labels than the same judge given a one-line instruction, because the levels reduce how much the model has to infer about your intent.
Test the rubric before you trust it
A rubric is a specification, and specifications have bugs. The cheapest test is to have two people independently score the same 30 to 50 examples and compare. Where they disagree, the rubric is ambiguous, not the reviewers. Fix the level definitions, add the disputed case as a worked example, and run it again.
Only after humans agree with each other should you hand the rubric to a judge model. Then measure the same way: how often does the judge match the human label. That number is the honest accuracy ceiling of any eval built on this rubric.
Expect to revise. New failure modes appear, and an edge case turns out to sit exactly on a level boundary. Version the rubric when that happens, because scores produced under version 1 and version 2 are not the same measurement.
Rubrics for agents
Grading an agent means grading a sequence, and that changes what the rubric describes. Instead of “is this answer correct,” the criteria cover whether the agent selected an appropriate tool, passed valid arguments, recovered after a failed call, avoided redundant work, and stopped when it had enough information. Each of those is a separate dimension with its own levels.
This is where the rubric has to be written against the trace rather than the response. Scoring behavior inside an agent harness with its tools and controls means the evaluator needs access to the intermediate steps, and the rubric has to say which steps count as evidence.
Coding agents are the clearest case. A rubric for one might cover whether the change compiles, whether tests pass, whether the diff stays inside the requested scope, and whether the agent explained what it changed. The first two are mechanical, the last two need judgment, and a trace-level LLM evaluation guide generally runs both kinds together, alongside LLM as a judge practices for the open-ended dimensions.
FAQ
What does rubric mean in AI?
It means the same thing it means in education: a written set of criteria and levels used to grade work consistently. In AI evaluation, the work being graded is a model output or an agent trajectory, and the grader is a human reviewer, an LLM judge, or both.
What is the difference between a rubric and a scoring function?
The rubric is the human-readable criteria: the dimension, the levels, and what evidence counts. The scoring function is the mechanism that applies those criteria and emits a value, whether that is a judge prompt, a piece of code, or an annotation queue. One rubric can be applied by several scoring functions, and comparing their outputs is a good way to find out whether your criteria are as clear as you think.
How many levels should a rubric have?
Two for pass/fail checks, three or four for graded quality. Beyond that, the added granularity is usually noise. If you need finer resolution, split the dimension into two rubrics rather than stretching the scale.
Can an LLM write the rubric?
It can produce a reasonable first draft, and that is a fine way to avoid a blank page. It cannot decide what your product considers acceptable, which cases are policy violations, or which tradeoff matters to your users. Draft with the model, then correct it against real failures you have read yourself.
How do I know if my rubric is working?
Two signals. Agreement between independent reviewers on the same examples should be high, and the score should move when the system’s behavior actually changes. A rubric that produces the same number regardless of what you ship is measuring something other than your system.
