An evaluation pipeline is the repeatable workflow that takes evaluation inputs, runs scoring logic, stores results, and triggers follow-up actions. Inputs might be production traces, curated datasets, test cases, sessions, or agent trajectories. Scoring might use deterministic checks, LLM judges, embedding metrics, human labels, or custom functions.
A good evaluation pipeline is reproducible. Developers should be able to answer: what was evaluated, which evaluator version ran, which model or prompt version produced the output, what changed since the last run, and which failures need action.