The data and datasets layer is the part of an evaluation system that stores and serves the examples used for testing, scoring, experimentation, and improvement. It holds production traces, curated test sets, golden datasets, human labels, synthetic examples, and the metadata attached to all of them.
Calling it a layer is a claim about architecture, not about content. Your application emits telemetry, your evaluators produce scores, and something in the middle has to hold the examples both of those refer to. That middle piece decides whether a score recorded in March can still be compared to one recorded in September, and whether a failing eval case can be traced back to the exact production request it came from.
Better evaluators cannot compensate for a dataset that does not represent real failure modes. But that is a question about what you put in. This page is about where it lives and how it is wired.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- The layer sits between the trace store and the evaluation runtime. Traces flow in as candidate examples, datasets flow out into experiment runs, and scores flow back onto both.
- A dataset row is more than an input and an output. It carries an example ID, a dataset version, metadata for slicing, and a pointer back to the
trace_idorspan_idit was promoted from. - Datasets are versioned snapshots rather than mutable tables. Without a version stamped on each run, two scores from different weeks are not comparable and no dashboard work fixes that.
- Record every score with the dataset version, task version, and evaluator version that produced it. Those three stamps are what make a regression explainable.
- The common failure is architectural: a CSV on someone’s laptop, no version history, no link back to the originating trace, and metadata dropped at collection time.
What the layer holds
Four kinds of records with different mutability and different lifetimes.
Traces and spans. Append-only telemetry from the running application. A span records one operation with its inputs, outputs, timing, and attributes; a trace is the tree of spans for one request. This is raw material rather than a dataset, and it usually lives under a retention window because volume scales with traffic.
Datasets and examples. Curated collections that persist independently of retention. An example is one row: an input, optionally a reference output or the criteria a judge should apply, metadata such as source and segment and failure category, and the trace pointer it came from. For agents the useful row often carries the full trajectory rather than a single input and answer.
Annotations and labels. Human judgments attached to a span or an example, stored as separate records rather than columns. The same example can carry labels from several reviewers at different times, and you want to keep the disagreement rather than average it away at write time.
Experiment and evaluation runs. The output side. A run references a dataset version, the task version it exercised, the evaluator configuration, and one result per example. Runs are immutable once complete, which is what lets you compare them.
The connections that make it a layer
Three joins do all the work, and each one is an ID.
Trace to example. Promotion is the act of taking a real production record and adding it to a dataset. The pointer back to the originating span is the part teams forget, and it is the part you want six months later when a case fails and you need to know what the system actually did on that request.
Example to run. An experiment run iterates over a dataset version, executes the task for each example, and writes one result per row. Because the run pins the version, comparing two runs compares the thing you changed rather than two different datasets. Datasets also have practical size ceilings in any platform, and those limits move, as larger dataset support and evaluation changes in a release history will show.
Score back to record. Evaluation results are written onto the span or example they scored, not into a separate spreadsheet. That is what makes a score filterable next to everything else you know about the request, and it is why trace and session level evaluation is a storage decision as much as a metric decision. A score with no record attached can be reported but not debugged.
Where the layer breaks
No versions. Someone adds twelve hard cases to the regression set, the average drops four points, and nobody can tell whether the system got worse or the test got harder. This is the most common and most expensive failure in the layer.
Broken lineage. The dataset was assembled by hand from exported rows, so the trace pointer is gone. Debugging a failing case now means guessing.
Metadata dropped at write time. Segment, language, source, and failure category cost seconds to record at promotion and are miserable to reconstruct. Without them you can report an aggregate but you cannot slice it.
Governance as an afterthought. Production traces contain whatever your users typed, which means personal data, credentials, and customer content. The layer needs retention rules, redaction on ingest, and access control that matches how teams are separated. Ordinary data quality checks apply here too: nulls, type mismatches, and truncated fields corrupt an eval set as happily as they corrupt a feature table.
FAQ
Is the datasets layer the same thing as a data warehouse or feature store?
No, though they share ideas. A warehouse is organized for analytics and a feature store is organized for serving features at low latency. The datasets layer is organized around examples and their lineage: which run used which version, and which trace each example came from. Teams often keep the underlying bytes in a warehouse and still need the example, version, and run structure on top.
Where should datasets live relative to traces?
Close enough to join cheaply, separate enough to outlive them. Traces expire under a retention window; a curated example should not. The practical arrangement is a telemetry store with retention plus a dataset store without it, linked by span and trace IDs, so promoting a record copies what you need and keeps the pointer.
How do you version a dataset?
Treat changes as new versions rather than edits in place, and stamp the version onto every run that reads it. You need the ability to say “this score came from version 7” and to re-run version 7 later. Whether that is snapshots, an append log, or tagged commits matters much less than never mutating a set that historical scores depend on.
Does this layer make sense before you have production traffic?
Yes, and it is cheaper to set up first. Pre-launch you will have handwritten cases and synthetic examples rather than promoted traces, but the structure is identical: examples with IDs, versions, metadata, and runs that reference a version. The layer then fills in from real traffic instead of being retrofitted around a folder of spreadsheets.
Who owns this layer in practice?
Usually whoever owns the evaluation suite, which in most teams is the application engineers rather than a separate data team. The part needing an explicit owner is the corpus of reference answers, because those expire when policy changes and an unowned reference answer quietly becomes a wrong one.
