Skip to main content

Runnable Python app

Generate sample traces with OpenAI and run the FastAPI service that connects AX to Jev.

​
Overview

In this tutorial, you will create customer-support traces in Arize AX, then configure a remote evaluator that asks TypeSafe Jev whether each response resolves the user’s request. Jev’s Noul question returns a probability that the answer is yes. The evaluator maps 0.5 or higher to yes, and anything lower to no. It returns that probability as the score. The companion app calls a live OpenAI model (gpt-5.4-mini by default). Its six requests mix general support questions with account-specific actions that the demo agent has no tools to perform. The agent gets the same capability context for every request; no example tells it to produce a pass or fail. This mix is intended to show both Jev labels, but live responses and judgments can vary, so a 3/3 split is not guaranteed.

​
Before you start

You will need:
  • An Arize AX account, an AX API key, and a Space ID. Remote evaluators are currently available to Enterprise accounts.
  • An OpenAI API key with access to the selected model.
  • A TypeSafe API key.
  • Python 3.10 or later.
  • A tunneling tool such as Cloudflare Tunnel or ngrok to expose your local evaluator to AX during the demo.
  • The example app from Arize-ai/tutorials.

​
Generate support traces

The traces you generate here are the ones you will evaluate with the Jev remote evaluator later in this guide.
1

Clone the tutorials and configure the apps

Clone Arize-ai/tutorials. From the directory containing the cloned tutorials folder, run:
The agent and evaluator each have a separate .env.example with only their required values. The agent configuration sets the AX project name; set OPENAI_MODEL in agent/.env to override the default model.
2

Generate the sample traces

Run the agent:
The agent sends six baked-in support requests to OpenAI. The requests mix general questions with account-specific actions that the demo agent cannot perform without account tools. The model is not told to pass or fail individual examples, so its live responses and Jev judgments can vary. Open the project in AX and confirm the traces are present before continuing.

​
Configure and run the Jev evaluator

The evaluator is a FastAPI service. AX sends it the mapped request and response fields; the service forwards them to Jev with the question Does the response resolve the user’s request? and returns a label and score.
1

Install and start the evaluator

In a second terminal, start from the same directory containing the cloned tutorials folder, then install the evaluator dependencies and start the service with its TypeSafe key:
2

Check the evaluator health

In another terminal, confirm the service loaded the TypeSafe key:
3

Expose the evaluator with your tunnel

Use your tunnel to provide AX with an HTTPS endpoint it can reach. For example, start a temporary Cloudflare quick tunnel in a third terminal:
Copy the HTTPS URL provided by your tunnel. If you use Cloudflare Tunnel, the quick-tunnel URL looks like https://<random-name>.trycloudflare.com. The unauthenticated quick tunnel is intended only for this temporary demo. Stop it when finished.

​
Create the remote eval

1

Create Remote Eval

In AX, open Evaluators → New Evaluator → Create Remote Eval.
2

Set the name and scope

Name the evaluator Jev evaluator and set its scope to Span.
3

Set the endpoint

Set the endpoint to your forwarded HTTPS endpoint with /v1/evaluate appended. For example, when using Cloudflare Tunnel, the endpoint looks like https://<random-name>.trycloudflare.com/v1/evaluate. This demo endpoint does not require headers.
4

Set the input schema

Set the input schema to:
5

Set the data source and span filter

Set the data source to jev-remote-evaluator-balanced, the project name in agent/.env.example. When you select the project, AX automatically maps input to attributes.input.value and output to attributes.output.value. Filter spans by kind so only LLM spans are included.
6

Test the endpoint on a span

Choose one of the generated spans, then select Test Remote On Span. A successful result includes a yes or no label and a score from 0 to 1. Save the evaluator after confirming the response.
7

Run the evaluator on the project

Set the task name to Run Jev Evaluator. Turn on Run Continuously and One-Time Backfill, then select Create and Run Evaluator to create the evaluator task and start it. Open a sample trace and confirm the evaluation result is attached.

​
Understand the result

Jev returns a probability for the Noul question. The service uses that probability directly as score and applies a 0.5 threshold for the label: For example, a result of {"label":"yes","score":0.91} means Jev estimates a 91% probability that the response resolves the request. This is a judge estimate, not a guarantee that the underlying action occurred.

​
Summary

You generated support traces, connected an AX Remote Eval to TypeSafe Jev, tested the endpoint on a span, and ran the evaluator against the project. The OpenAI response and Jev judgment are live model outputs and can vary.

​
Next steps

Remote evaluators

Learn about the request and response contract, endpoint behavior, and other hosting options.

Run evals on your traces

Learn more about configuring evaluation tasks for trace data.