Evaluation as infrastructure means treating evaluation as a core system dependency, in the same category as logging, tracing, testing, and CI/CD. It is not a periodic review or a spreadsheet of examples somebody maintains on the side. It is a shared layer that development workflows, production monitors, agents, and humans can all call.
The claim behind the phrase is that evaluation is a platform capability rather than a project activity. A project activity ends. Someone runs an eval before a launch, writes up the numbers, and the script rots in a notebook. A platform capability persists: any team can score a new service without building anything first, and any score from six months ago can still be traced back to the dataset, evaluator, and judge model that produced it.
As infrastructure, evaluation needs APIs, datasets, versioning, runners, storage, permissions, monitors, and action hooks. The value is not the score by itself. The value is the workflow the score triggers.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Evaluation as infrastructure is the argument that evals are a shared platform layer, not a per-project script. The test is whether a new service can get its first eval running without new infrastructure work.
- The layer has concrete parts: trace storage, a versioned dataset store, an evaluator registry, runners for CI and for sampled production traffic, an API, permissions, monitors, and hooks that turn a score into an action.
- Infrastructure is what you build; EvalOps is how you run it. The components are the same either way, and the operating practice sits on top.
- Reproducibility is the point of the versioning. Every stored score should name its dataset version, evaluator version, and judge model version, or you cannot compare across time.
- The score is not the deliverable. If nothing happens when a score crosses a threshold, you have built reporting rather than infrastructure.
What the layer actually contains
- Trace and span storage. Production evals read what the system did, so instrumented traces are the substrate. This is the same data that supports tracing and evaluating agents for debugging, which is why the two capabilities usually get built together rather than separately.
- A dataset store with versions. Named, immutable dataset versions so a score is attached to a specific set of rows. Without versions, “the eval suite” is a moving target.
- An evaluator registry. Judge prompts, code evaluators, parsing rules, and thresholds as versioned artifacts rather than strings pasted into scripts.
- Runners. At least three execution modes: batch over a dataset, invocation from CI on a change, and sampled scoring of live production spans.
- An API. Programmatic invocation so CI jobs, notebooks, internal tools, and agents can all run the same evaluator and get comparable results.
- Permissions and tenancy. Who can see traces, who can edit an evaluator, how teams stay isolated in a shared platform.
- Monitors and action hooks. The part that makes it infrastructure instead of a report: a score crossing a threshold opens an alert, blocks a merge, files a review task, or queues a rerun.
The through-line is that these are the same components regardless of scale. What the runner does is what a project-level script does. The difference is that the platform version is shared, versioned, and callable, which is roughly the difference between an evaluation harness and a one-off evaluation script.
Why project-based evaluation stops working
Four failure patterns show up as soon as more than one team is shipping AI features.
Nobody can reproduce a number. A score from last quarter came from a notebook with an unpinned judge model and a dataset that has since been edited. There is no way to know whether today’s number is better.
Scores are not comparable across teams. Two teams each wrote a “correctness” judge. The prompts differ, one uses a five-point score and the other a binary label, and a leadership dashboard averages them anyway.
Every team rebuilds the plumbing. Loading traces, batching calls, handling rate limits, parsing judge output, storing results. That work is identical everywhere and it is not where the domain knowledge is.
Findings do not persist. A production incident gets diagnosed, fixed, and forgotten, because there is nowhere durable to put the failing case as a permanent test.
Each of these is an infrastructure gap rather than an evaluation gap, which is the argument for a single place where evaluation and observability live together instead of a set of disconnected scripts and dashboards.
Where evaluation as infrastructure ends and other terms begin
- EvalOps is the operating practice on top of this layer: ownership, cadence, gating rules, alert routing, and review. Infrastructure is the thing you build. EvalOps is how you run it.
- Evals as APIs is the interface property of this layer, meaning evaluation is callable by code rather than only clickable in a UI.
- Observability provides the traces evaluation reads and the monitoring surface evaluation results feed. Evaluation adds the judgment about whether behavior was good.
What to build first
The minimum useful version is: traces stored somewhere queryable, one versioned dataset built from real traffic, one evaluator whose labels you have compared against human judgment, a runner you can call from CI, and one alert with a named owner. That is small enough to build in a sprint and it already supports the loop. Tenancy, a registry, and monitor-triggered workflows are worth adding when more than one team depends on it.
FAQ
How is evaluation as infrastructure different from EvalOps?
Infrastructure is the set of components: storage, datasets, evaluators, runners, an API, monitors. EvalOps is the practice of operating them: who owns each evaluator, how often suites run, what gates a release, who responds to an alert. Having the infrastructure and no practice is the common state, and the result is a platform that produces numbers nobody acts on.
Do we need a platform team to do this?
No, but you do need someone to own the shared parts. Small teams get most of the value by standardizing three things: where traces live, how datasets are versioned, and where evaluator definitions are stored. A dedicated team becomes worthwhile when several product teams are running evals and you want their results to be comparable.
Is this just testing with extra steps?
It overlaps with testing and differs in two ways. The oracle is often a model rather than an assertion, so results are distributions with thresholds instead of equality assertions. And much of the evaluation happens after release, on production traces, which is not something a test suite does.
What is the minimum viable eval infrastructure?
Queryable traces, one versioned dataset drawn from real traffic, one validated evaluator, a runner callable from CI, and one alert with an owner. Everything else is an extension of that core.
How does it relate to tracing and observability?
Tracing produces the record of what happened; evaluation judges whether what happened was acceptable. They share storage and a data model, and evaluation results are most useful when they are attached to the specific span or trace that produced them, so a bad score leads directly to the example behind it.
