An evaluation store, also sometimes referred to as an inference store, is a machine learning infrastructure tool used to monitor and improve model performance. Think of it as the ledger or log of model activities and inferences. It persists predictions, inputs, metadata, labels when they arrive, and derived metrics so teams can aggregate performance, compare environments, and explain regressions without re-scoring raw logs by hand each time.
Key takeaways
- An evaluation store records inference events and links them to ground truth, scorer outputs, and model version metadata.
- It powers aggregate and sliced metrics in production, validation, and training replay workflows.
- Baselines stored alongside current traffic make drift and performance degradation visible early.
- The same store supports A/B comparisons and post-incident forensics across model versions.
- Treat retention, PII, and schema evolution as first-class design choices, not afterthoughts.
What gets stored
Typical records include request identifiers, timestamps, model name and version, input features or prompt text (often redacted or hashed), model outputs (labels, scores, embeddings summaries), latency, and optional tags for cohort analysis. When labels arrive later, the store joins them back to the original prediction for delayed metric computation.
Some teams also store evaluator outputs: LLM judge scores, rule-based check results, and human annotation IDs. That turns the store into the system of record for both classical ML metrics and generative quality signals.
Core use cases
Aggregate and slice metrics. Roll up accuracy, error rate, or custom eval scores by day, region, or customer tier. Slices surface where a model fails even when global metrics look flat.
Drift and data quality monitoring. Compare current input distributions and prediction distributions to a baseline window stored in the same system. Spikes in missing features or embedding distance often precede metric cliffs.
Connect performance changes to causes. When accuracy drops, analysts filter to the affected slice, inspect feature shifts, and correlate with deploy events or upstream pipeline changes.
Continuous delivery feedback. Compare production metrics to offline validation and training replay on recent traffic samples. Mismatch flags incomplete eval sets or serving skew.
Experimentation. Route traffic between model versions and read outcomes from one store so A/B tests do not require duplicate pipelines.
Architecture patterns
Ingestion paths include batch exports from the serving layer, streaming events from OpenTelemetry spans, or periodic pulls from warehouse tables. Storage may be columnar (for analytics), object storage (for large payloads), or a hybrid with hot indexes for recent windows.
Schema design matters. Version feature names, embedding model IDs, and prompt template hashes as explicit fields so historical queries stay valid after refactors. Partition by time and model to bound query cost.
Access control separates raw inputs (often sensitive) from aggregated metrics shared broadly. Many teams store hashed user IDs and keep plaintext prompts in restricted tables with shorter retention.
Relationship to observability and eval harnesses
Tracing captures request-level spans; an evaluation store emphasizes metric-ready tables and label joins. The two complement each other: traces debug one failure; the store quantifies how often failures happen.
An evaluation harness defines scorers and datasets offline. The evaluation store applies the same scorer definitions (or compatible ones) to production samples so offline and online numbers mean the same thing.
LLM and agent evaluation platforms often bundle ingestion, evaluators, and dashboards. Whether you buy or build, the store remains the durable layer that survives individual dashboard refactors.
Failure modes
Incomplete joins. Labels never link to predictions because IDs differ between logging and label pipelines. Metric jobs silently drop rows.
Schema drift. New features log under renamed columns; historical slices break. Enforce schema contracts at ingest.
Retention gaps. Incidents discovered after TTL expiry lack raw rows for replay. Tier retention by regulatory need and debugging value.
Double counting. Retries and duplicate events inflate volume metrics. Deduplicate on stable request IDs.
PII leakage. Storing full prompts or payloads without policy blocks enterprise adoption. Redact at ingest.
Operating the store in practice
Define golden queries: daily accuracy by model, top ten worst slices, label latency histograms. Alert on metric deltas against baselines, not on single bad rows.
Replay recent production traffic through candidate models offline using stored inputs. Writing LLM evals as tests extends that pattern: export a failing cohort from the store into CI fixtures so fixes prove themselves before redeploy.
FAQ
Is an evaluation store the same as a feature store?
No. A feature store serves transformed inputs for training and inference. An evaluation store records outputs, labels, and metrics after inference. Pipelines may write to both, but the responsibilities differ.
Do I need an evaluation store for batch-only models?
Yes, if you re-score on a schedule or need historical comparisons. Batch scoring jobs can append rows the same way online serving does. Without a store, each analysis becomes a one-off notebook.
How is this different from a data warehouse?
Warehouses hold broad business data. An evaluation store optimizes ML-specific joins: predictions, model versions, eval scores, and monitoring baselines. Many teams land evaluation store tables in the warehouse for SQL access while keeping hot paths specialized.
What retention should I use?
Match legal requirements and debugging needs. Thirty to ninety days of row-level detail plus longer aggregated rollups is a common split. Keep enough raw data to reproduce incidents you discover within your SLA.
Can one store cover LLM agents and tabular models?
Yes, with flexible payload columns and typed metadata. Unify on request ID, model version, and timestamp; let task-specific fields vary by schema version rather than forcing one flat table for all modalities.
