Improve Your Agent

In the previous guide, the groundedness evaluator revealed a pattern: the chatbot makes claims not in the policy documents - the system prompt says “be helpful” but doesn’t enforce grounding. Rather than guessing at a fix and redeploying, start from a real failure, fix it in Playground using the exact inputs that went wrong, then validate across a full dataset before shipping.

Arize AX skyserve-chatbot Traces view with Alyx assistant open, user request about groundedness-check failures this week, and Alyx task plan and progress

This is Part 3 of the Arize AX Get Started series. You should have completed the Evaluations guide first, with evaluation scores visible on your traces.

Choose how you want to work

Use Arize Skills to have your coding agent run improvement workflows from your editor, Alyx for a conversational approach inside the Arize platform, the UI for a hands-on step-by-step experience, or Code to run programmatically. In each path, you’ll build a dataset from failing traces, iterate on your prompt, and compare experiments before shipping.

By Arize Skills
By Alyx
By UI
By Code

Use Arize Skills with your coding agent to run the same workflow from your editor. The example prompts below are what you type to your agent — the skill loads automatically and handles the rest. Install the skills plugin and follow Set up Arize with AI coding agents for authentication and CLI setup.

Step 1: See evaluation results on your traces

arize-traceSpans include labels once an eval task has run; see Viewing results in the tracing UI.For example, you might say:

Export spans from skyserve-chatbot where groundedness-check failed this week

Terminal showing ax spans export command, export success message, summary of spans with low groundedness flagged as hallucinated, and a table of span and trace IDs with evaluator columns

Step 2: Create a dataset

arize-datasetFor example, you might say:

Create skyserve-test-cases from those failing traces

Terminal: dataset skyserve-test-cases created from failing traces, with schema fields question, reference_text, original_output, trace and span IDs, and status counts

Step 3: Improve the system prompt

arize-prompt-optimizationFor example, you might say:

Extract the system prompt from the failing skyserve-chatbot spans and generate an improved version. Use the groundedness-check eval labels and explanations as signal for what to fix.

Improved SkyServe system prompt with grounding rules and notes referencing groundedness-check eval labels and span-level failures

Step 4: Run both prompts as experiments

arize-experimentReuse the same evaluators you trust in production; see Run evals on experiments.For example, you might say:

Run both prompt versions (original and the updated one) against the dataset and compare groundedness scores.

Experiment comparison table: skyserve-original-prompt at 4/5 groundedness versus skyserve-improved-prompt at 5/5 (100%)

Follow these steps in the Arize AX UI: find failing traces, replay them in the Prompt Playground, tighten your prompt, build a dataset, run experiments, compare evals, and save to Prompt Hub.

Step 1: See evaluation results on your traces

Go to your skyserve-chatbot project and filter traces by the groundedness-check evaluation score. Find a trace that failed — one where the chatbot made up information not in the policy documents — and click in to see what went wrong.

Traces filtered by groundedness evaluation showing hallucinated traces

Step 2: Replay in Prompt Playground

Click Open in Playground on the span. AX automatically populates the system prompt, user message, and model settings that produced the bad answer — no manual setup needed.

Trace detail for an LLM ChatCompletion span showing trace tree, span evaluations, Open in Playground, and Input Output tab with model and system prompt

Prompt Playground auto-populated from a trace with system prompt, user message, and model

Step 3: Improve the system prompt

The original is too loose:

You are SkyServe Airlines' customer service assistant.
Answer the customer's question based on the provided policy documents.
Be friendly and helpful.

Tighten it with explicit grounding rules — for example, require that every claim reference a specific policy, and instruct the model to say “I don’t have that information” rather than guess. Click Run to confirm the response improves.

Playground with improved system prompt and new grounded response

Step 4: Create a dataset and run experiments

A few spot-checks aren’t enough. Create a dataset of representative test cases (common questions, edge cases, known failures) and run both prompt versions against it as experiments — one baseline, one improved. In Datasets, add examples (upload a CSV or build from traces) and open the dataset in Playground. Run your original prompt as the baseline experiment, then run your improved prompt on the same inputs.

New Dataset dialog showing CSV upload with preview of test cases

Experiments tab showing baseline-original-prompt experiment

Playground with improved prompt and dataset ready to run

Step 5: Evaluate and compare

Add your groundedness-check evaluator to both experiments (the same one you created in the Evaluations guide) and use Compare mode to view results side by side. You can add a Helpfulness evaluator from the templates to check that answers stay useful. You should see groundedness improve while helpfulness stays flat. If you see a regression, iterate in Playground.

Add Evaluator flow showing available evaluators from the hub

Compare Experiments view showing two experiments side by side

Step 6: Save to Prompt Hub

Once satisfied, click Save to Prompt Hub, name it skyserve-support, and add a version description. Your prompt is now versioned — your team can see the full history, compare versions, and roll back if needed.

Save to Prompt Hub dialog with name, description, and version description

Prompt Hub showing skyserve-support version history and prompt template

Run this workflow from the Python SDK, TypeScript SDK, or ax CLI. Some features are in alpha or beta - please check individual reference pages for details.

Step	Python SDK	TypeScript SDK	CLI
Filter spans by eval result	Link	Link	Link
Create a dataset from traces	Link	Link	Link
Manage prompts	Link	Link	Link
Run experiments	Link	Link	Link

Congratulations!

You’ve completed the full improvement loop:

Traced your app to see what’s happening inside it.
Evaluated responses automatically to measure quality.
Improved your prompt using real failure data in the Playground.
Proved the improvement works across a representative dataset with experiments.

You now have a repeatable, data-driven process for improving your LLM application. No more guessing, no more hoping - you can measure quality and demonstrate improvement. Next up: Deepen your tracing foundation so your improvement loop stays grounded in complete, high-quality telemetry.

Choose how you want to work