An evaluation pipeline is the repeatable workflow that takes evaluation inputs, runs scoring logic, stores results, and triggers follow-up actions. Inputs might be production traces, curated datasets, test cases, sessions, or agent trajectories. Scoring might use deterministic checks, LLM judges, embedding metrics, human labels, or custom functions.
The word pipeline is doing real work here. It describes data flow: a sequence of stages, each with an input shape and an output shape, that turns raw model activity into scores you can query. That is a different concern from the runner that executes a single evaluation. The harness invokes the system and applies scorers. The pipeline is the path data travels before and after that step, including how cases get selected, where results land, and how they get joined back to the trace that produced them.
A good pipeline is reproducible. You should be able to answer, months later: what was evaluated, which evaluator version ran, which model or prompt version produced the output, what changed since the last run, and which failures need action. If any of those answers requires guessing, the pipeline is missing a field.
Key takeaways
- A pipeline is data flow, not a runner. Selection, preparation, execution, scoring, persistence, aggregation, and action are separate stages that fail separately.
- Every stored score needs the versions that produced it: dataset, prompt, model, scorer, and harness. Without them, two runs are not comparable.
- Scoring runs asynchronously off traces rather than inside the request path, so evaluation latency never becomes user-facing latency.
- Sampling is a design decision, not an afterthought. Scoring all production traffic with a judge model is usually not affordable, so decide what gets sampled and record the rate.
- Errors and low scores must be distinguishable in storage. A scorer that crashed is not a model that failed.
The stages
Selection. Something decides which records enter the pipeline: a whole curated dataset for an offline run, or a sampled slice of production traffic for an online one. Selection logic is the most common place bias enters, because a filter written to catch interesting cases will overrepresent them.
Preparation. Raw traces are not evaluation inputs. Getting there means pulling the right spans out of a trace, reassembling the retrieved context, flattening a multi-turn session into something a scorer can read, and attaching whatever reference answer exists. The mechanics of collecting agent traces and evaluating them determine how much of this stage you have to build yourself.
Execution. The system under test runs, or, for online evaluation, it already ran and you are scoring what it produced. This is where the harness lives.
Scoring. Scorers run against the prepared input. Deterministic checks are cheap and should run first. Judge calls are expensive and often run only on records that passed the cheap checks or landed in a suspicious slice. Human labels arrive on a slower loop, and building annotation into the pipeline is what keeps the automated scorers anchored to something real.
Persistence. Scores are written with their metadata: the record ID, the scorer name and version, the score, the label, the explanation if the scorer produced one, the timestamp, and the run ID. This is the stage teams shortchange, and it is the one that determines whether the pipeline can answer questions later.
Aggregation. Individual scores become a run summary: pass rate, mean and quantiles of the score distribution, results per slice, and the delta against a baseline run. Slice results matter more than the headline number, because an average holds steady while one segment breaks.
Action. The run either does something or it does not exist. Actions include failing a build, alerting on a slice, routing failures to a review queue, opening a ticket, or seeding the next experiment.
Offline and online paths
Most teams end up with two pipelines that share stages. The offline path runs a fixed dataset on demand: pre-merge, pre-release, or whenever a prompt changes. It answers whether this version is better than that version. The online path samples live traffic continuously and answers whether quality is holding in production right now.
They differ in what they can tell you. Offline runs have reference answers and a stable comparison baseline, so they support gating decisions. Online runs have real inputs and no ground truth, so they lean on reference-free scorers such as groundedness checks and on user signals such as retries, escalations, and thumbs-down. Building both, and knowing which one you trust for which decision, is roughly the middle of the maturity curve from a first eval to ongoing AI operations.
Where pipelines break
Cost growth nobody planned for. Judge calls scale with traffic times sample rate times number of scorers. Three scorers on 10% of a busy endpoint is a large recurring bill, and it arrives quietly.
Silent stage failures. If preparation cannot find the retrieved context, a groundedness scorer will confidently score against an empty document set. The pipeline keeps reporting numbers.
Unversioned scorers. Editing a judge prompt to improve it makes every prior score incomparable to every future one. Treat scorer prompts as versioned artifacts.
Backfill that cannot happen. A new scorer is only useful on history if you kept the inputs. Retaining the prepared evaluation input, not just the score, is what makes re-scoring last quarter possible.
Drift in the set itself. A dataset assembled six months ago stops resembling current traffic. The pipeline still runs, still passes, and stops telling you anything.
FAQ
What is the difference between an evaluation pipeline and an evaluation harness?
The pipeline is the sequence of stages data moves through to become a stored score. The harness is the software that executes one of those stages: it loads cases, invokes the system under test, and applies scorers. Teams often say pipeline when they mean the whole apparatus, which is fine in conversation, but the distinction matters when you are debugging. A missing score is usually a pipeline problem. A wrong score is usually a harness or scorer problem.
Should evaluation run inside my application request path?
No. Scoring adds model calls, and putting them inline makes users wait for your measurement. The standard pattern is to emit traces from the application, then have the pipeline consume those traces asynchronously. The exception is a guardrail that has to block a response before it reaches the user, which is a different mechanism with a latency budget of its own.
How do I make an evaluation pipeline reproducible?
Pin and record five things with every result: the input dataset version, the prompt version, the model and its exact version string, the scorer version, and the harness version. Then store the prepared input alongside the score. Reproducibility fails most often because a provider updated a model behind a stable alias and nothing in your records shows it.
How much production traffic should the pipeline sample?
Enough for the slices you care about to be populated, which is usually a lower rate than teams expect for high-volume routes and a higher rate for rare ones. Sampling uniformly at 1% will leave your smallest important segment with almost nothing in it, so stratify by segment and record the per-segment rate so aggregates can be weighted correctly later.
