Evaluation gating is the use of eval results to allow, block, or require review before a change moves forward. A gate might block a prompt update if task success rate drops, if policy adherence fails, or if retrieval relevance regresses on a golden dataset. It is the mechanism that connects a score to a decision, and without it an eval suite is a report that ships alongside the regression.
The gate is a workflow object, not a metric. It has a location in your release process, a set of criteria it evaluates, an outcome it produces, and a rule about who can override it. Get those four things explicit and gating works. Leave any of them implicit and you end up with a check that everyone has learned to re-run until it goes green.
Gating is closely related to pass/fail criteria and it helps to keep them apart. The criteria are the specific thresholds that decide pass or fail. The gate is where those criteria run and what happens to the release when they fail.
Key takeaways
- A gate needs four defined parts: where it runs, which criteria it checks, what outcome it produces, and who can override it.
- Gate on the delta against a baseline, not on an absolute score. Absolute thresholds either block constantly or never fire.
- Tier your gates. Deterministic checks on every commit, a sampled judge suite per pull request, the full suite nightly and before release. Running everything on every commit does not scale.
- Judge nondeterminism makes gates flaky. Size the sample and the tolerance band using measured run-to-run variance, not intuition.
- A gate with no override path gets disabled. Provide a documented escape hatch with a named approver and an audit record.
Where gates live
Pre-merge, in CI. The gate runs on a pull request that touches a prompt, a tool definition, a retrieval configuration, or model selection. It has to be fast, because a check that takes 25 minutes becomes a thing people merge around. This tier is deterministic assertions plus a small sampled judge suite.
Pre-deploy, in the release pipeline. The full offline suite runs against the release candidate and compares to the current production version. Expensive scorers belong here, since this runs less often and can afford a higher cost per run. It is still worth knowing how evaluation costs accumulate across a suite before putting a thousand judged cases in a blocking step.
During rollout. The change goes to a small share of traffic and online evals score the live results before it goes further. This gate has no reference answers, so it uses reference-free scorers and product signals: task completion, retry rate, escalation, safety violations. It needs the production scoring path already running, which is one reason teams stand up an observability platform for agents before attempting progressive delivery.
At runtime. Worth naming so it does not get conflated: a guardrail that blocks or rewrites an individual response is not evaluation gating. It runs per request, inside the latency budget, and gates output rather than a release.
What a gate outputs
Three outcomes, not two: allow, block, and require review. The third one is what keeps a gate usable. Most eval movements are neither clearly fine nor clearly broken, and a binary gate forces every one of those into a category that is wrong. Route the ambiguous band, plus any run where a scorer errored, to a person with the failing cases and the judge’s explanation attached.
Making the decision correctly
Compare to a baseline. The useful question is whether this candidate is worse than what is running now, on the same cases with the same scorer version. Absolute thresholds age badly: set at 0.8, they block every experiment early on and pass everything once the system improves.
Separate must-pass from tolerance. Safety and compliance assertions block on a single failure. Aggregate quality criteria get a tolerance band, because a small drop is usually noise. “Fail if any P0 safety test fails or if task success drops more than 3 percent against baseline” is a gate. “Average score below 0.8” is a wish.
Account for scorer noise. Judge models return different scores on identical inputs, so a gate on a small suite will fail builds for no reason. Measure variance by running the same candidate several times, then set the tolerance outside that band. This is a standing cost of judge-based scoring and part of deciding where an LLM judge is appropriate versus where a deterministic check does the job more cheaply.
Gate on slices. An aggregate that holds while one segment collapses is the most common way a gate misses a real regression.
Why gates get bypassed
A gate that fails often for reasons the team does not believe will get routed around, and the routing around is informal: re-running until it passes, merging with an override, or commenting out the step during a deadline. That is worse than no gate, because it produces a record saying the check ran.
Three practices keep it honest. Fix flakiness rather than raising the threshold to accommodate it, since raising it hides real regressions too. Make the override require a named approver and leave an audit trail, so exceptions are countable instead of invisible. And review the criteria on a cadence, because a threshold set once and never revisited stops describing the bar the team actually holds.
FAQ
What is an evaluation gate in CI?
A required check on a pull request that runs evals against the proposed change and fails the build if criteria are not met. Mechanically it is the same as a test gate: the CI job runs the harness on a fixed dataset, compares results to a stored baseline for the target branch, and exits non-zero on failure. The difference from unit tests is that results are scores with variance rather than booleans, so the gate compares distributions and needs a tolerance.
Should a gate block or just warn?
Block for anything you would roll back for, primarily safety, compliance, and hard output contract violations. Warn for early-stage quality metrics where you have not yet measured the noise band, and promote them to blocking once you have. Starting everything as a warning is a reasonable way to avoid teaching the team to ignore the gate on day one, as long as the promotion actually happens.
Which evals belong in a pre-merge gate versus a nightly run?
Pre-merge gets what is fast, cheap, and deterministic, plus a small judge suite on the cases most likely to break. Nightly and pre-release get the full suite, the expensive judges, multi-turn trajectory evaluations, and anything that needs repeated runs to stabilize. The split is driven by wall-clock time and cost, not by importance.
How do I handle a flaky gate caused by judge variance?
Measure the variance before adjusting anything: run the same candidate through the suite several times and look at the spread. Then reduce it, by using a stronger or more constrained judge, tightening the rubric, averaging repeated scores per case, or replacing the judge with a deterministic check where one exists. Only after that, set the tolerance outside the remaining noise. Widening the tolerance first is how gates become theater.
