An evaluation harness is the operational system that turns evals into repeatable workflows and actions. Concretely, it is the software that loads a set of cases, invokes the system under test on each one, applies scorers to the output, and collects the results somewhere you can compare them later.
A harness is more than a single evaluator or judge prompt. A judge prompt scores one output. A harness decides which outputs get scored at all, runs them under controlled conditions, records the version of everything involved, and does something with the result. Without one, you have a set of evals that someone runs by hand and a score nobody can reproduce two weeks later.
The part that surprises people: the harness is part of what you are measuring. The same model, given the same task, scores differently depending on the retry policy, the tool schemas, how much context gets assembled, the step limit, and how the output is parsed. When a number moves, the harness is one of the first suspects.
Key takeaways
- A harness has three moving parts: the inputs it evaluates, the execution that produces scores, and the actions it takes afterward.
- The harness is a variable in the experiment. Change the scaffolding and the same model produces different scores, so the harness has to be versioned like any other dependency.
- Deterministic checks belong in the fast path and expensive judges belong in a slower tier, because scoring every case with a judge model on every run gets costly quickly.
- Most harness bugs read as model regressions. A parse failure scored as zero looks exactly like a model that got worse.
- A harness is the runner. The pipeline is the sequence of stages it moves data through, and gating is what happens to a release when the scores come back.
The three parts of a production harness
Evaluation inputs define what gets evaluated: traces, spans, agent trajectories, sessions, production examples, or curated offline datasets. This is where the harness either represents your traffic or quietly stops representing it.
Evaluation execution defines how scoring happens. It can use LLM judges, deterministic checks such as schema validation or string assertions, embedding similarity, custom scorers, human review, or agent-based evaluation. Execution also covers the unglamorous mechanics: concurrency, timeouts, retries, and isolating one case from the next.
Evaluation actions define what happens next. Results can trigger alerts, route examples to an annotation queue, block a deployment, open a ticket, start an experiment, or feed failures back into the next round of prompt work. A harness that produces a number and stops is a report generator.
Why the harness is part of the measurement
Swap the model behind an agent and hold everything else constant, and you learn something about the model. Swap the scaffolding and hold the model constant, and you learn that your scores were partly about the scaffolding all along. Comparisons that hold one agent harness fixed and vary the model depend on that: the harness has to be the constant, or the comparison measures both at once.
That has a practical consequence. Every harness change needs a version, and every stored result needs to carry the version that produced it. Otherwise you cannot tell whether last month’s 0.82 and this month’s 0.76 are comparable at all. The architecture and controls that make up an agent harness are the same surface you have to pin down before an eval number means anything.
Cost, latency, and what does not scale
Running the full suite with a judge model on every commit does not scale. Each case costs at least one extra model call, often several, and a few hundred cases with multiple scorers turns into real money and real minutes. Teams that ignore this either stop running evals or start ignoring the results.
The usual arrangement is tiered. Deterministic checks such as schema validity, required-field presence, and forbidden-string assertions run on every commit in seconds. The core judge suite runs per pull request on a sampled set. The full suite runs nightly or before a release. Budgeting that means knowing where the money in running evaluations actually goes, because judge calls are only part of it.
Failure modes worth checking first
- Scorer errors counted as failures. A judge that times out or returns unparseable JSON should be recorded as an error, not a score of zero. Conflating them makes infrastructure problems look like quality regressions.
- State leaking between cases. Shared conversation memory, a warm cache, or a record left behind by case 4 changes the result of case 5, and then sequential and parallel runs disagree.
- Version skew. The prompt, model, retrieval index, scorer, and harness can all move. If any is unpinned, a score change has no single explanation.
- Nondeterminism read as signal. Judge models return different scores on identical inputs. A one-point move on a 50-case suite is usually noise, and only repeated runs tell you the size of the noise band.
- A set that no longer matches traffic. The harness keeps passing while production changes. That is evaluation drift, the failure mode behind green dashboards and a worse product.
FAQ
What are the three stages of a production LLM evaluation harness?
Inputs, execution, and actions. Inputs determine what gets evaluated, whether that is a curated dataset or sampled production traces. Execution determines how the score is produced, using judges, deterministic checks, or human review. Actions determine what the score changes: an alert, a blocked deploy, a queued example for review, or a new experiment.
Is an evaluation harness the same thing as an evaluation pipeline?
They overlap, and the useful split is runner versus data flow. The harness is the software that executes a run: it loads cases, calls the system, applies scorers. The pipeline is the sequence of stages data moves through to become a stored, queryable score, including sampling, preparation, persistence, and aggregation. A harness is usually the execution stage inside a larger pipeline.
Do I need a harness if I already have unit tests?
Yes, because the outputs are graded differently. Unit tests assert exact behavior and return true or false. LLM outputs vary between runs and are usually graded on a scale by a scorer that is itself imperfect, so you need repeated runs, aggregate thresholds, and per-slice results rather than a single pass or fail per case. A harness handles that; a test runner alone does not.
Should the harness use the same model for scoring that it uses for the task?
Prefer not to, and if you do, check the effect. A model asked to grade its own output can favor its own phrasing and reasoning style, which inflates scores in a way that will not show up as a failure anywhere. Using a different judge model, or a small panel of them, is the common mitigation.
