You can’t read every output your AI system produces, so at some point you hand the grading to another LLM. This makes sense, but the trouble is where it tends to go next: teams point a judge at things like JSON validation, latency checks, and rules a regex would settle in milliseconds. The need to scale review is real. But not every task is a job for an LLM judge, and this article is about how to tell which ones are.
The signal you need a judge: subjective criteria code cannot express
A code-based evaluator answers one kind of question: did the output satisfy a deterministic condition? Did the JSON parse? Did the response stay under 200 tokens? Did the tool call reference the correct field name? If your question has that shape, write the check in code.
The question changes shape when quality depends on meaning. “Is this response helpful?” cannot be expressed as a boolean. “Did the agent’s reasoning steps follow a logical path to the conclusion?” has no regex. These signals call for an LLM judge: criteria like helpfulness, tone, groundedness, or reasoning quality that code cannot directly test.
The case for judging agentic systems is sharper. A research agent that calls three tools and synthesizes a response might produce the right final answer through flawed reasoning. Code that checks only the final output misses that entirely. As Beyond the Final Answer and AgentProcessBench both show, evaluation that looks only at final task outcomes skips reasoning steps, tool use, and intermediate logic. Checking only for success or failure misses how and why the agent got there.
What has to be true first: a clear rubric, the right context, categorical labels
A rubric written in plain language with examples
The rubric is the job description for the judge. Without one, models invent scoring criteria mid-evaluation and produce inconsistent outputs that tell you nothing about underlying quality. Write the criterion, define what a passing response looks like, define what a failing response looks like, and give one concrete example of each. The rubric does not need to be elaborate. It needs to be unambiguous.
The right context passed to the judge
An LLM judge takes four inputs: the original prompt, the candidate output, an optional reference answer, and the rubric. For subjective quality dimensions like tone or helpfulness, you can often omit the reference answer. For correctness tasks, the reference answer does heavy lifting. No Free Labels demonstrated a strong connection between an LLM’s ability to correctly answer a question and its ability to grade responses to that question. Providing the judge with a human-written reference answer improves agreement with human annotators, even for weaker judge models. If you are asking whether a factual claim is accurate, give the judge the correct answer to compare against.
Categorical labels, not numeric scores
Our LLM-as-a-judge guide separates the two: LLM judges handle semantic evaluations; code handles deterministic checks. Within semantic evaluations, the label format matters. Numeric scores on a 1-5 scale feel precise, but they’re not. The difference between a 3 and a 4 is undefined, so different judge runs will interpret it differently. Binary or categorical labels (Pass/Fail, or Grounded/Partially Grounded/Hallucinated) tie the score to a defined state. They are auditable and translate directly to business rules.
Require the judge to explain its label before assigning it Our research on prompting strategies shows that requiring an explanation alongside a label cuts variance across repeated judgments and increases agreement with human annotators. Explanation-first prompting functions as calibration built into the prompt.
The boundary case worth naming: for factual correctness tasks (math, code execution, schema validation, structured data extraction), an LLM judge should not be your first check even when you have a reference answer. A compiler or an interpreter does not hallucinate. Reach for the deterministic verifier first, and reserve the judge for the semantic dimensions those verifiers cannot reach.
When not to reach for a judge: deterministic checks, numeric scoring, tasks you cannot calibrate
Three conditions disqualify a task from LLM judging:
- A deterministic check already exists. Validating a JSON schema, matching a regex, or running a unit test produces a correct answer without inferring it probabilistically. Running an LLM over those tasks adds latency and cost while introducing the possibility of incorrect verdicts.
- The scoring format is numeric. Uncalibrated scales produce noisy data that accumulates in dashboards without telling you how to fix anything. Convert to categorical labels before deploying.
- No golden dataset exists to validate against. Without human-labeled examples to check the judge’s behavior, you are deploying a judge you cannot trust. Position bias, verbosity bias, and self-preference bias are common risks in uncalibrated systems (Gu et al., 2024). Without a golden dataset, you cannot detect whether any of these biases are distorting your scores.
Calibrate the judge against human labels before you trust it
Start calibration by assembling a golden dataset of 50 to 200 human-labeled examples. Cover the range of quality your system produces: clean successes, edge cases, and failures. Run the judge against every example. Compare the judge’s labels to the human labels.
Use Cohen’s kappa as the inter-rater metric rather than raw percent agreement, because kappa discounts the agreement you would get by chance. Future AGI’s 2026 evaluation benchmarks set the bar: above 0.6 is acceptable, above 0.8 is production-ready. If you fall below those thresholds, the rubric has a gap. Revise the rubric instead of the judge model.
Also choose the judge model deliberately. Using the same model family to judge its own outputs creates measurable self-preference. Our self-evaluation bias experiment found that an OpenAI judge scored its own outputs well above the baseline it assigned to other models (a +9.4 effect), with Anthropic showing a smaller but highly stable skew (+4.27). Use a different model family for the judge than for the generator, or accept that your scores carry a known upward skew.
Run judges continuously on production traces, not just offline
Models update, prompts change, and real user inputs deviate from anything in your test distribution. A judge that scores accurately in development can degrade when production data shifts, so the same evaluator must run in both environments. Metrics are only comparable when they come from the same judge on the same rubric. Running a different judge in production than in development is the same as changing the ruler between measurements.
Cost is often the objection to continuous judging. Frontier-tier judges run 50 to 100 times cheaper than human reviewers; smaller flash-tier models run 500 to 1,000 times cheaper. At those rates, covering production traffic with a judge works for most teams.
For clinical, legal, or financial applications, continuous judging does not replace human review. It scopes it. The judge acts as a first-pass filter, and human reviewers handle the traces that fall below the threshold.
Implementing continuous evaluation with Arize AX
With Arize AX Online Evals, you attach a calibrated judge to live production traces so the same evaluator that ran against your golden dataset during calibration also handles real traffic. The Evaluator Hub stores versioned, reusable evaluators. The same rubric runs in development, at the pre-release stage, and in production, so scores stay comparable across the lifecycle. That consistency is what makes improvement detectable: when a prompt change shifts the judge score in prod, you know the rubric did not change underneath it.
For teams starting offline, Phoenix Evals provides the open-source path. Run the judge against a dataset, measure agreement against human labels, and promote the evaluator to production once it clears the threshold. See the evaluator types guide for a practical decision guide on when to layer code checks alongside LLM judges.
Pick one trace type where your agents fail most often: detecting hallucinations, grading RAG relevance, or verifying tool calls. Write the rubric. Assemble 50 labeled examples. Run calibration. Deploy only after the agreement threshold clears.
Qualifying your next LLM judge deployment
Volume alone does not justify a judge. The task has to qualify.
Three questions before the next deployment decision: Can the criterion be written in plain language? If not, the rubric does not exist yet, and the judge has nothing to score against. Can 50 or more human-labeled examples be assembled? If not, the judge cannot be trusted. Does a deterministic check already exist? If yes, reach for the check.
The three questions function as a gate, rather than a checklist. A judge that clears all three is ready for calibration. One that fails any of them does not belong in the evaluation pipeline yet. The prerequisites exist to prevent noise from entering a system where it compounds quietly and gets mistaken for signal.
FAQs about when to use LLM as a judge
How do I choose between a frontier model and a smaller model for judging?
Frontier models provide the reasoning depth needed for calibrating rubrics because they handle complex reasoning with higher accuracy. Once a rubric is stable and agreement with human labels is high, you can often distill that logic into a flash-tier model to reduce costs significantly.
What should I do when human annotators disagree on a golden dataset label?
Disagreement usually signals an ambiguous rubric rather than a poor judge. If two experts cannot agree on a label, the criteria are too subjective for an LLM to score consistently. Resolve these conflicts by refining the rubric with a concrete example of the “tie-breaker” scenario before using the example for calibration.
Can LLM judges evaluate code generation or is a test runner better?
A test runner or compiler is always better for verifying if code executes, passes unit tests, or follows a schema. Use an LLM judge only for the semantic dimensions a compiler cannot see, such as whether the code follows specific style guidelines or if the logic matches the user’s intent.
How many examples does a golden dataset need to be trustworthy?
Assembling 50 to 200 human-labeled examples provides enough variety to calculate a meaningful Cohen’s kappa score. Smaller datasets often miss edge cases like verbosity bias or position bias, which can lead to overconfidence in the judge’s production performance.
How do I detect when a production judge has drifted?
Monitor the distribution of categorical labels in production and compare them against your calibration baseline. If the “Pass” rate shifts significantly without a corresponding change in the application, or if the judge’s explanations become shorter, it may indicate model drift or a shift in user input patterns that requires a rubric update.