Feature Store Monitoring

Feature store monitoring puts continuous, automated checks on the features flowing through your store so you catch pipeline shifts, distribution changes, and train-serve skew before they degrade model performance. Because feature stores fail silently, the root cause of a bad model often traces back to the feature data itself, and monitoring the data directly is the most reliable way to catch it early. For background on feature stores and why teams monitor them, see Feature Store: What’s All the Fuss?. In Arize AX, you snapshot your offline materializations and online serving logs, ingest them on a cadence, and attach drift and data quality monitors to each feature. Monitors advance their baselines automatically across deployments, run as a background process, and route alerts to wherever your team triages incidents.

How it works

Feature store monitoring on Arize AX: snapshots feed a continuous monitoring engine that combines drift, data quality, and online/offline alignment, then routes alerts.

Ingestion: snapshot your feature store

Arize ingests materialized features through whichever path fits your stack:

SDK (Python/Java) when you want to push snapshots from your own pipeline code.
File importer for object storage such as S3, GCS, or Azure Blob (Parquet/CSV).
Table importer for warehouse query layers such as Databricks, BigQuery, or Snowflake.

Feature stores are large and high-throughput, so monitoring is snapshot-based rather than streaming every row:

Take periodic snapshots of your feature store, covering both offline materializations (the values used for training) and online serving logs (the values used at inference).
Sample the snapshot down to a representative size so monitoring stays fast and cost-effective.
Ingest on a cadence that matches your materialization schedule, setting model_version to the snapshot timestamp so each ingestion lands as a distinct, comparable window of feature data.

Arize is built for ML model observability, so it assumes every row carries a prediction. Feature store snapshots usually have no model output, so you must include a prediction column and give it a value (a constant dummy such as 0 works). A null or missing prediction will error out on ingestion.

Upload Data to Arize

Log directly via SDK

File importer (S3 example)

Table importer (Databricks)

Auto-advancing baselines

Drift and data quality are measured against a baseline distribution. As you ship new feature versions and redeploy pipelines, that baseline would normally drift out of date and require manual reconfiguration. Arize advances baselines automatically. Monitors track to the latest baseline version on their own, so you do not have to reconfigure them between deployments. Once configured, the monitoring engine runs as a background process and stays quiet until an alert fires, which means coverage keeps up with a fast-moving feature store without ongoing babysitting. See drift monitor baselines for the baseline options and the baseline glossary entry for definitions.

Implicit checks via drift

Drift detection catches what explicit rules cannot. You do not have to know in advance which feature will break or how, because drift compares the current distribution of each feature to its baseline and surfaces any meaningful shift:

Silent pipeline shifts, where a transformation changes behavior without erroring.
Upstream distribution changes, where a vendor or source feed moves underneath you.
Feature skew between the online and offline serving paths.

In practice, drift does more of the work than explicit rules once a team is up and running, because it generalizes to failures nobody wrote a rule for. Arize offers several distributional metrics (PSI, KL divergence, JS distance, KS statistic) so you can tune sensitivity to your tolerance for false positives.

Drift Tracing

Drift metrics reference

Explicit data quality checks

Explicit data quality monitors enforce the rules you do know, scoped to a single snapshot:

Null-rate thresholds, so a feature that suddenly goes empty trips an alert.
Range guards, so values outside expected bounds are flagged.
Schema validation, so cardinality changes, missing values, and new categories are caught.

These are precise and easy to reason about, but they only cover the failure modes you anticipated. Drift is the layer on top, not a replacement: explicit checks guard the known and structural failures per snapshot, while drift catches the distributional shifts you did not enumerate.

Troubleshoot Data Quality

Data quality metrics reference

Online/offline alignment

Train-serve skew is one of the most damaging and hardest to spot feature store problems: the value a model trained on differs from the value it sees at serving time. Arize lets you compare serving-time feature values against batch materializations directly and surface that skew as a monitor instead of a manual audit. Ingest both the online serving snapshot and the offline materialization snapshot for the same window, then set up day-over-day alignment monitoring on the features. This catches divergence between the two code paths as it develops, rather than discovering it after a model has been quietly underperforming in production. For more on why online/offline feature skew happens and how monitoring addresses it, see Feature Store: What’s All the Fuss?.

Set it up programmatically

The fastest way to get coverage is Arize’s managed monitors: once a model is receiving data, Arize automatically creates drift and data quality monitors with auto-calculated thresholds, so a large feature set gets baseline coverage without writing any configuration. Managed monitors are a great starting point, then you create custom monitors when you need per-feature control or want to reduce noise on a large store. For control-plane integration, you create those custom monitors programmatically via GraphQL, REST, or the Python SDK, so feature store checks become system-driven decisions rather than manual UI workflows, with evaluation frequency as granular as every 5 minutes. In all cases, lead with auto thresholding. Arize sets the threshold from your historical data (mean or median based) and it adapts as features evolve. Static thresholds are an option when you already know the exact level you want to alert on.

1. Ingest a feature snapshot

Log a sampled snapshot of your feature store with the Python SDK to the production environment. Set model_version to the snapshot timestamp so each ingestion is a distinct, comparable window. Because Arize expects a prediction on every row, include a dummy prediction column with a constant value (a null or missing prediction errors on ingestion). Add a tag that marks which serving path the snapshot came from, so you can scope alignment monitors to online vs offline later.

from arize.pandas.logger import Client
from arize.utils.types import Environments, ModelTypes, Schema

client = Client(space_id="YOUR_SPACE_ID", api_key="YOUR_API_KEY")

# snapshot is a sampled dataframe of feature values. Arize requires a prediction
# per row, so add a constant dummy column if your features have no model output:
#   snapshot["prediction"] = 0
#   snapshot["serving_path"] = "online"  # or "offline"
schema = Schema(
    prediction_id_column_name="row_id",
    timestamp_column_name="materialized_at",
    prediction_label_column_name="prediction",
    feature_column_names=["txn_amount", "account_age_days", "merchant_category"],
    tag_column_names=["serving_path"],
)

client.log(
    model_id="feature-store-fraud",
    model_version="2026-06-08T00:00:00",  # snapshot timestamp
    model_type=ModelTypes.SCORE_CATEGORICAL,
    environment=Environments.PRODUCTION,
    dataframe=snapshot,
    schema=schema,
)

Log your online serving snapshot and your offline materialization snapshot the same way, setting serving_path to "online" or "offline" on each. With both in production you can filter alignment monitors by that tag.

2. Create a data quality monitor

Create an explicit null-rate guard on a single feature with GraphQL. Lead with an auto threshold so Arize sets the alert level from historical data. A static threshold (for example threshold: 0.02) is an option if you already know the exact level you want.

mutation {
  createDataQualityMonitor(input: {
    modelId: "model_id"
    name: "txn_amount null rate"
    dimensionCategory: featureLabel
    dimensionName: "txn_amount"
    dataQualityMetric: percentEmpty
    operator: greaterThan
    dynamicAutoThreshold: {
      stdDevMultiplier: 2
    }
    contacts: [{
      notificationChannelType: email,
      emailAddress: "you@me.com"
    }]
  }) {
    monitor { id }
  }
}

3. Create a drift monitor

Create an implicit drift check with PSI. Again, lead with the auto threshold so it adapts as the feature evolves. A static threshold is an option if you prefer to set the level yourself.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    name: "txn_amount drift"
    dimensionCategory: featureLabel
    dimensionName: "txn_amount"
    driftMetric: psi
    operator: greaterThan
    dynamicAutoThreshold: {
      stdDevMultiplier: 2
    }
    baseline: {
      useModelPrimaryBaseline: true
    }
  }) {
    monitor { id }
  }
}

You can loop these mutations over every feature in a snapshot to get exhaustive coverage, and re-run them in CI as new features land. See Programmatically Create Monitors for query, patch, baseline, and filter options.

Alert routing

When a monitor fires, Arize routes the alert to wherever your team works. Supported destinations include PagerDuty, Slack, OpsGenie, email, AWS EventBridge, Airflow, and generic webhooks. That covers both human-in-the-loop triage (page an on-call engineer, post to a channel) and fully automated remediation (trigger a backfill, gate a deployment, kick off a retrain through your control plane).

Notifications and integrations

Why drift plus data quality cover it

The combination is what makes this work:

Explicit data quality checks guard the known and structural failures, per snapshot: nulls, out-of-range values, schema and cardinality changes.
Drift detection catches the unknown, distributional shifts no one wrote a rule for, and in practice does most of the work once teams are running.
Online/offline alignment catches train-serve skew directly, as a monitor instead of a manual audit.

Together, with auto-advancing baselines and snapshot-based ingestion, they give you continuous feature store coverage that keeps up with fast-moving pipelines, without manual reconfiguration or after-the-fact audits.

Additional resources

Monitors overview

Programmatically Create Monitors

Drift Tracing

Troubleshoot Data Quality

Blog: Feature Store, What's All the Fuss?

Upload Data to Arize

Need help setting up feature store monitoring? Reach out via email at support@arize.com or join us in the Arize community Slack.

Quickstart

Instrument

Observe

Evaluate

Improve

Agents

Machine Learning

Settings

Security

Feature Store Monitoring

How it works

Ingestion: snapshot your feature store

Upload Data to Arize

Log directly via SDK

File importer (S3 example)

Table importer (Databricks)

Auto-advancing baselines

Implicit checks via drift

Drift Tracing

Drift metrics reference

Explicit data quality checks

Troubleshoot Data Quality

Data quality metrics reference

Online/offline alignment

Set it up programmatically

1. Ingest a feature snapshot

2. Create a data quality monitor

3. Create a drift monitor

Alert routing

Notifications and integrations

Why drift plus data quality cover it

Additional resources

Monitors overview

Programmatically Create Monitors

Drift Tracing

Troubleshoot Data Quality

Blog: Feature Store, What's All the Fuss?

Upload Data to Arize

​How it works

​Ingestion: snapshot your feature store

Upload Data to Arize

Log directly via SDK

File importer (S3 example)

Table importer (Databricks)

​Auto-advancing baselines

​Implicit checks via drift

Drift Tracing

Drift metrics reference

​Explicit data quality checks

Troubleshoot Data Quality

Data quality metrics reference

​Online/offline alignment

​Set it up programmatically

​1. Ingest a feature snapshot

​2. Create a data quality monitor

​3. Create a drift monitor

​Alert routing

Notifications and integrations

​Why drift plus data quality cover it

​Additional resources

Monitors overview

Programmatically Create Monitors

Drift Tracing

Troubleshoot Data Quality

Blog: Feature Store, What's All the Fuss?

Upload Data to Arize

How it works

Ingestion: snapshot your feature store

Auto-advancing baselines

Implicit checks via drift

Explicit data quality checks

Online/offline alignment

Set it up programmatically

1. Ingest a feature snapshot

2. Create a data quality monitor

3. Create a drift monitor

Alert routing

Why drift plus data quality cover it

Additional resources