Python SDKBlogSlack

Configure Monitors

Learn about the different configurations users can customize for monitors

This guide will help you set up and configure monitors to track your data and get alerts when something goes wrong. Whether you're new to monitoring or need to adjust existing monitors, this guide covers everything you need to know.

What You'll Learn

  • Types of Monitors

  • When Monitors Run (Evaluation Windows)

  • Setting Alert Thresholds

  • Choosing Your Baseline Data

  • Getting Notified When Issues Occur

Types of Monitors

Arize offers two types of monitors to help you track your data:

Managed Monitors

  • What they are: Monitors that Arize creates automatically for you

  • Best for: Getting started quickly with basic monitoring

  • How to configure: Go to the 'Monitors' tab → 'Managed Performance Monitors' card

  • Important: Changes here affect ALL managed monitors of the same type

⚠️ Warning: For features or drift, especially on large feature sets, they can be noisy because you're mass producing alerts and not taking each feature's characteristics into account

Custom Monitors (Recommended)

  • What they are: Monitors you create yourself with specific settings

  • Best for: Ensuring you are monitoring the things you need and reduces noise and ensures the proper configuration

  • How to configure: Each monitor has its own settings in "Step 2: Define the Data"

  • Important: Each custom monitor can have completely different settings

💡 Tip: Start with managed monitors to get familiar with monitoring, then create custom monitors when you need more control.

When Monitors Run (Evaluation Windows)

Think of evaluation windows as "how often should I check my data?" Here's what you need to know:

Evaluation Window (How Often to Check)

  • What it is: How frequently your monitor looks at your data

  • Examples: Every hour, every day, every week

  • Why it matters: More frequent checks = faster alerts, but more notifications

Common Settings:

  • Hourly: For critical systems that need immediate attention

  • Daily: For most business applications (recommended)

  • Weekly: For less critical monitoring

Delay Window (When to Start Checking)

  • What it is: How long to wait before starting to monitor

  • Why you need it: Prevents false alerts when your data comes in batches or is delayed

  • Example: Wait 2 hours after new data arrives before checking for problems

💡 Tip: Set a delay window if you're getting false alerts. This gives your system time to stabilize.

API-Triggered Monitors

  • What it is: A monitor type that only evaluates when triggered via API call, instead of on a fixed schedule.

  • Why it’s useful: Ideal for teams that run evaluations after specific events, such as:

    • Batch ingestion

    • Completion of a model retraining pipeline

    • Custom evaluation workflow or CI/CD step

How it works:

  • You create and configure the monitor as usual.

  • Instead of choosing an hourly, daily, or weekly schedule, select “Manual".

  • Arize will only evaluate the monitor when your system makes an API call to the monitor’s endpoint.

To run the monitor, use the following mutation:

    mutation TriggerMonitorMutation {
        triggerMonitor(input: {monitorId: "MONITOR_ID"}) {
            success
            monitor {
                name
                id
            }
        }
    }

Setting Alert Thresholds

Thresholds are like setting the "alarm level" for your monitor. When your data crosses this line, you get an alert.

Threshold Ranges

What are Threshold Ranges?

Define acceptable value limits for monitors, offering control over alert conditions.

Why Use Them?

  • Flexibility: Set both upper and lower bounds.

  • Reduced Alerts: Focus on significant deviations.

  • Precision: Target specific performance issues.

Two Ways to Set Thresholds

1. Automatic Thresholds

  • What it is: Arize automatically figures out a good threshold based on your historical data

  • Best for: Most users who want to get started quickly

  • How it works: Looks at your past data and sets a reasonable alert level

  • Need to define: The two types of auto threshold mean and median and when to use each

2. Static Thresholds

  • What it is: You set the exact number where you want alerts

  • Best for: When you know exactly what values are "good" vs "bad"

  • Example: "Alert me if accuracy drops below 85%"

  • Also missing: The ability to have a threshold range

Adjusting Automatic Thresholds

For all managed monitors at once:

  1. Go to 'Monitors' tab

  2. Click 'Config'

  3. Find 'Managed Performance Monitors' card

  4. Adjust the sensitivity slider

For individual monitors:

  1. Go to 'Monitors' page

  2. Click 'Monitors Listing'

  3. Select your monitor

  4. Go to 'Define the Alerting' section

Understanding Sensitivity

Sensitivity Levels and Deviations

  • High Sensitivity: Triggers alerts on smaller deviations (e.g., 1-2 standard deviations from the mean).

  • Medium Sensitivity: Suitable for most cases, alerts on moderate deviations (e.g., around 2-3 standard deviations).

  • Low Sensitivity: Only alerts for large deviations (e.g., beyond 3 standard deviations).

💡 Tip: Start with medium sensitivity, adjust if necessary—lower if there are too many alerts, higher if important issues are missed.

Setting Your Baseline

For all monitors:

  1. Go to 'Dataset' or 'Config' tab

  2. Click 'Configure Baseline'

  3. Choose your baseline data

For individual monitors:

  1. Edit your monitor

  2. Go to "Step 2: Define the Data"

  3. Select 'Custom Baseline'

  4. Choose your specific baseline

💡 Tip: Start with the default baseline. Only change it if you have a specific reason to compare against different data.

Getting Notified When Issues Occur

Once your monitor detects a problem, you need to know about it! Here's how to set up notifications.

Monitor Status Explained

Your monitor will always be in one of these states:

  • 🟢 Healthy: Everything looks good! No action needed

  • 🟡 No Data: Your monitor hasn't received recent data to check

  • 🔴 Triggered: Something looks wrong - you should investigate

Setting Up Notifications

When your monitor turns red (triggered), you can get notified through:

  • Email: Get alerts in your inbox

  • Slack: Get alerts in your team's Slack channel

  • PagerDuty: For critical systems that need immediate attention

  • OpsGenie: For teams that use OpsGenie for incident management

Two Ways to Set Notifications

1. Bulk Settings (For All Managed Monitors)

Best for: Setting up notifications for all your basic monitors at once

  1. Go to 'Monitors' tab

  2. Click 'Config'

  3. Find the 'Notifications' section

  4. Add your email, Slack channel, or other notification methods

2. Individual Monitor Settings

Best for: When you want different notifications for different monitors

  1. Go to 'Monitors' page

  2. Click 'Monitors Listing'

  3. Select the monitor you want to edit

  4. Go to 'Step 4: Define the Notification'

  5. Add your notification preferences

Need Help?

  • Getting too many alerts? Lower your threshold sensitivity or check your delay is set right

  • Missing important issues? Raise your threshold sensitivity

Common Mistakes to Avoid

  1. Setting up monitors for every feature/attribute - Only notify for things that really matter

  2. Using static thresholds without understanding your data - Start with automatic thresholds

  3. Changing too many settings at once - Start simple and adjust gradually

Last updated

Was this helpful?