Python SDKBlogSlack

Configure Monitors

Learn about the different configurations users can customize for monitors

This guide will help you set up and configure monitors to track your data and get alerts when something goes wrong. Whether you're new to monitoring or need to adjust existing monitors, this guide covers everything you need to know.

What You'll Learn

  • Types of Monitors

  • When Monitors Run (Evaluation Windows)

  • Setting Alert Thresholds

  • Choosing Your Baseline Data

  • Getting Notified When Issues Occur

Types of Monitors

Arize offers two types of monitors to help you track your data:

Managed Monitors

  • What they are: Monitors that Arize creates automatically for you

  • Best for: Getting started quickly with basic monitoring

  • How to configure: Go to the 'Monitors' tab → 'Managed Performance Monitors' card

  • Important: Changes here affect ALL managed monitors of the same type

⚠️ Warning: For features or drift, especially on large feature sets, they can be noisy because you're mass producing alerts and not taking each feature's characteristics into account

Custom Monitors (Recommended)

  • What they are: Monitors you create yourself with specific settings

  • Best for: Ensuring you are monitoring the things you need and reduces noise and ensures the proper configuration

  • How to configure: Each monitor has its own settings in "Step 2: Define the Data"

  • Important: Each custom monitor can have completely different settings

💡 Tip: Start with managed monitors to get familiar with monitoring, then create custom monitors when you need more control.

When Monitors Run (Evaluation Windows)

Think of evaluation windows as "how often should I check my data?" Here's what you need to know:

Evaluation Window (How Often to Check)

  • What it is: How frequently your monitor looks at your data

  • Examples: Every hour, every day, every week

  • Why it matters: More frequent checks = faster alerts, but more notifications

Common Settings:

  • Hourly: For critical systems that need immediate attention

  • Daily: For most business applications (recommended)

  • Weekly: For less critical monitoring

Delay Window (When to Start Checking)

  • What it is: How long to wait before starting to monitor

  • Why you need it: Prevents false alerts when your data comes in batches or is delayed

  • Example: Wait 2 hours after new data arrives before checking for problems

💡 Tip: Set a delay window if you're getting false alerts. This gives your system time to stabilize.

Setting Alert Thresholds

Thresholds are like setting the "alarm level" for your monitor. When your data crosses this line, you get an alert.

Two Ways to Set Thresholds

1. Automatic Thresholds

  • What it is: Arize automatically figures out a good threshold based on your historical data

  • Best for: Most users who want to get started quickly

  • How it works: Looks at your past data and sets a reasonable alert level

  • Need to define: The two types of auto threshold mean and median and when to use each

2. Static Thresholds

  • What it is: You set the exact number where you want alerts

  • Best for: When you know exactly what values are "good" vs "bad"

  • Example: "Alert me if accuracy drops below 85%"

  • Also missing: The ability to have a threshold range

Adjusting Automatic Thresholds

For all managed monitors at once:

  1. Go to 'Monitors' tab

  2. Click 'Config'

  3. Find 'Managed Performance Monitors' card

  4. Adjust the sensitivity slider

For individual monitors:

  1. Go to 'Monitors' page

  2. Click 'Monitors Listing'

  3. Select your monitor

  4. Go to 'Define the Alerting' section

Understanding Sensitivity

Think of sensitivity like a smoke detector:

  • High sensitivity = Alerts for small changes (more notifications, catches issues early)

  • Low sensitivity = Only alerts for big changes (fewer notifications, might miss small issues)

💡 Tip: Start with medium sensitivity. If you get too many alerts, lower it. If you miss important issues, raise it.

Need to give an idea of what the number of deviations is that equates for high and low

Choosing Your Baseline Data (Drift)

A baseline is like a "normal" reference point that your drift monitor compares new data against to understand if this distribution has shifted. Think of it as "what does good data look like?"

What is a Baseline?

  • What it is: A set of historical data that represents "normal" behavior

  • Why you need it: So your monitor knows when something is unusual

Types of Baselines

Default Baseline

  • What it is: Arize automatically picks a good baseline for you

  • Best for: Getting started quickly

  • How it works: Uses your recent data as the "normal" reference

Custom Baseline

  • What it is: You choose exactly which data to use as your baseline

  • Best for: When you have specific requirements

  • Examples:

    • Use data from last month as your baseline

    • Use data from a specific time period when things were working well

    • Compare against a different version of your system

Setting Your Baseline

For all monitors:

  1. Go to 'Dataset' or 'Config' tab

  2. Click 'Configure Baseline'

  3. Choose your baseline data

For individual monitors:

  1. Edit your monitor

  2. Go to "Step 2: Define the Data"

  3. Select 'Custom Baseline'

  4. Choose your specific baseline

💡 Tip: Start with the default baseline. Only change it if you have a specific reason to compare against different data.

Getting Notified When Issues Occur

Once your monitor detects a problem, you need to know about it! Here's how to set up notifications.

Monitor Status Explained

Your monitor will always be in one of these states:

  • 🟢 Healthy: Everything looks good! No action needed

  • 🟡 No Data: Your monitor hasn't received recent data to check

  • 🔴 Triggered: Something looks wrong - you should investigate

Setting Up Notifications

When your monitor turns red (triggered), you can get notified through:

  • Email: Get alerts in your inbox

  • Slack: Get alerts in your team's Slack channel

  • PagerDuty: For critical systems that need immediate attention

  • OpsGenie: For teams that use OpsGenie for incident management

Two Ways to Set Notifications

1. Bulk Settings (For All Managed Monitors)

Best for: Setting up notifications for all your basic monitors at once

  1. Go to 'Monitors' tab

  2. Click 'Config'

  3. Find the 'Notifications' section

  4. Add your email, Slack channel, or other notification methods

2. Individual Monitor Settings

Best for: When you want different notifications for different monitors

  1. Go to 'Monitors' page

  2. Click 'Monitors Listing'

  3. Select the monitor you want to edit

  4. Go to 'Step 4: Define the Notification'

  5. Add your notification preferences

Need Help?

  • Getting too many alerts? Lower your threshold sensitivity or check your delay is set right

  • Missing important issues? Raise your threshold sensitivity

Common Mistakes to Avoid

  1. Setting up monitors for every feature/attribute - Only notify for things that really matter

  2. Using static thresholds without understanding your data - Start with automatic thresholds

  3. Changing too many settings at once - Start simple and adjust gradually

Last updated

Was this helpful?