User Guide

Unified Platform across Development & Production

Arize AI provides an end-to-end platform designed to build, evaluate, and monitor LLM-powered applications at scale.

Development:

Curate and Manage Datasets

• Flexible Dataset Curation: Datasets can be created from spans, programmatically via code, or upload via CSV

• Leverage Production Data: Pull real-world interactions from production into datasets to refine prompts and responses

Run and Track Experiments

• Code-Based Experimentation: Run experiments programmatically to test different LLM models, retrieval strategies, and other parameters.

• Interactive Playground: Prototype and iterate on prompts and agent behaviors in the Prompt Playground before deployment.

• Prompt Hub: Store, version, and compare prompts in a central repository for better tracking and collaboration.

Evaluate Experiments

• Validate Experiment Outcomes: Ensure that modifications to prompts, models, or retrieval strategies result in measurable improvements.

• Compare Across Experiments: Standardize evaluation criteria to compare different iterations, models, or fine-tuned variants objectively.

CI/CD Every Change to the Application

• Automate CI/CD Workflows: Create experiments that automatically validate changes—whether it's a tweak to a prompt, model, or function — using a curated dataset and your preferred evaluation method.

Production

Trace & Debug Application Behavior

• Full LLM Trace Logging: Capture the entire LLM workflow, including function calls, retrieved context, chain-of-thought reasoning, and response outputs.

• Deep Debugging: Quickly identify latency, poor retrievals, prompt issues, understand agentic paths, replay voice assistants, and more.

Online Evaluation of Production Data

• Automated Real-Time Evaluation: Set up evaluation tasks that automatically tag new spans with evaluation labels as soon as the data arrives in the platform, streamlining the process.

• Track Application Outputs: Track degradation in performance, increased latency, or undesirable outputs (e.g., hallucinations, toxicity) on live application data.

• Annotations: Combine human expertise with automated workflows to generate high-quality labels and annotations.

Monitor and Protect Application

• Real-Time Application Monitoring: Set up alerts and dashboards to track LLM latency, token usage, evaluations, and more.

• Guardrails Against Poor Outputs: Automatically flag and prevent inappropriate, biased, or hallucinated responses before they reach users.