> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Jev as a Remote Evaluator

> Generate support traces in Arize AX, then evaluate whether each response resolves the request using TypeSafe Jev.

<Card title="Runnable Python app" href="https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/jev_remote_evaluator" icon="python" horizontal>
  Generate sample traces with OpenAI and run the FastAPI service that connects AX to Jev.
</Card>

## Overview

In this tutorial, you will create customer-support traces in Arize AX, then configure a [remote evaluator](/docs/ax/evaluate/remote-evaluators) that asks [TypeSafe Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) whether each response resolves the user's request. Jev's [Noul question](https://docs.typesafe.ai/primitives/noul) returns a probability that the answer is yes. The evaluator maps `0.5` or higher to `yes`, and anything lower to `no`. It returns that probability as the score.

The companion app calls a live OpenAI model (`gpt-5.4-mini` by default). Its six requests mix general support questions with account-specific actions that the demo agent has no tools to perform. The agent gets the same capability context for every request; no example tells it to produce a pass or fail. This mix is intended to show both Jev labels, but live responses and judgments can vary, so a 3/3 split is not guaranteed.

## Before you start

You will need:

* An [Arize AX account](https://app.arize.com), an AX API key, and a Space ID. Remote evaluators are currently available to Enterprise accounts.
* An OpenAI API key with access to the selected model.
* A [TypeSafe API key](https://console.typesafe.ai/).
* Python 3.10 or later.
* A tunneling tool such as [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) or [ngrok](https://ngrok.com/) to expose your local evaluator to AX during the demo.
* The example app from [Arize-ai/tutorials](https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/jev_remote_evaluator).

## Generate support traces

The traces you generate here are the ones you will evaluate with the Jev remote evaluator later in this guide.

<Steps>
  <Step title="Clone the tutorials and configure the apps">
    Clone [Arize-ai/tutorials](https://github.com/Arize-ai/tutorials). From the directory containing the cloned `tutorials` folder, run:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    cd tutorials/python/cookbooks/jev_remote_evaluator
    cp agent/.env.example agent/.env
    # Set the OpenAI and Arize values in agent/.env.
    cp evaluator/.env.example evaluator/.env
    # Set TYPESAFE_API_KEY in evaluator/.env.
    python -m venv agent/.venv
    source agent/.venv/bin/activate
    pip install -r agent/requirements.txt
    ```

    The agent and evaluator each have a separate `.env.example` with only their required values. The agent configuration sets the AX project name; set `OPENAI_MODEL` in `agent/.env` to override the default model.
  </Step>

  <Step title="Generate the sample traces">
    Run the agent:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    python agent/generate.py
    ```

    The agent sends six baked-in support requests to OpenAI. The requests mix general questions with account-specific actions that the demo agent cannot perform without account tools. The model is not told to pass or fail individual examples, so its live responses and Jev judgments can vary. Open the project in AX and confirm the traces are present before continuing.
  </Step>
</Steps>

## Configure and run the Jev evaluator

The evaluator is a FastAPI service. AX sends it the mapped request and response fields; the service forwards them to Jev with the question **Does the response resolve the user's request?** and returns a label and score.

<Steps>
  <Step title="Install and start the evaluator">
    In a second terminal, start from the same directory containing the cloned `tutorials` folder, then install the evaluator dependencies and start the service with its TypeSafe key:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    cd tutorials/python/cookbooks/jev_remote_evaluator
    python -m venv evaluator/.venv
    source evaluator/.venv/bin/activate
    pip install -r evaluator/requirements.txt
    cd evaluator
    set -a; source .env; set +a
    uvicorn remote_eval_server:app --host 127.0.0.1 --port 8080
    ```
  </Step>

  <Step title="Check the evaluator health">
    In another terminal, confirm the service loaded the TypeSafe key:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    curl http://127.0.0.1:8080/
    ```
  </Step>

  <Step title="Expose the evaluator with your tunnel">
    Use your tunnel to provide AX with an HTTPS endpoint it can reach. For example, start a temporary Cloudflare quick tunnel in a third terminal:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    cloudflared tunnel --url http://127.0.0.1:8080
    ```

    Copy the HTTPS URL provided by your tunnel. If you use Cloudflare Tunnel, the quick-tunnel URL looks like `https://<random-name>.trycloudflare.com`. The unauthenticated quick tunnel is intended only for this temporary demo. Stop it when finished.
  </Step>
</Steps>

## Create the remote eval

<Steps>
  <Step title="Create Remote Eval">
    In AX, open **Evaluators → New Evaluator → Create Remote Eval**.
  </Step>

  <Step title="Set the name and scope">
    Name the evaluator `Jev evaluator` and set its scope to **Span**.
  </Step>

  <Step title="Set the endpoint">
    Set the endpoint to your forwarded HTTPS endpoint with `/v1/evaluate` appended. For example, when using Cloudflare Tunnel, the endpoint looks like `https://<random-name>.trycloudflare.com/v1/evaluate`. This demo endpoint does not require headers.
  </Step>

  <Step title="Set the input schema">
    Set the input schema to:

    ```json theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    {
      "type": "object",
      "required": ["input"],
      "properties": {
        "input": {
          "type": "object",
          "required": ["input", "output"],
          "properties": {
            "input": {"type": "string"},
            "output": {"type": "string"}
          }
        }
      }
    }
    ```
  </Step>

  <Step title="Set the data source and span filter">
    Set the data source to `jev-remote-evaluator-balanced`, the project name in `agent/.env.example`. When you select the project, AX automatically maps `input` to `attributes.input.value` and `output` to `attributes.output.value`. Filter spans by kind so only **LLM** spans are included.
  </Step>

  <Step title="Test the endpoint on a span">
    Choose one of the generated spans, then select **Test Remote On Span**. A successful result includes a `yes` or `no` label and a score from `0` to `1`. Save the evaluator after confirming the response.
  </Step>

  <Step title="Run the evaluator on the project">
    Set the task name to `Run Jev Evaluator`. Turn on **Run Continuously** and **One-Time Backfill**, then select **Create and Run Evaluator** to create the evaluator task and start it. Open a sample trace and confirm the evaluation result is attached.
  </Step>
</Steps>

## Understand the result

Jev returns a probability for the Noul question. The service uses that probability directly as `score` and applies a `0.5` threshold for the label:

| Jev probability | Label | Score           |
| --------------- | ----- | --------------- |
| `0.5` or higher | `yes` | Jev probability |
| Below `0.5`     | `no`  | Jev probability |

For example, a result of `{"label":"yes","score":0.91}` means Jev estimates a 91% probability that the response resolves the request. This is a judge estimate, not a guarantee that the underlying action occurred.

## Summary

You generated support traces, connected an AX Remote Eval to TypeSafe Jev, tested the endpoint on a span, and ran the evaluator against the project. The OpenAI response and Jev judgment are live model outputs and can vary.

## Next steps

<CardGroup cols={2}>
  <Card title="Remote evaluators" icon="server" href="/docs/ax/evaluate/remote-evaluators">
    Learn about the request and response contract, endpoint behavior, and other hosting options.
  </Card>

  <Card title="Run evals on your traces" icon="gauge" href="/docs/ax/evaluate/run-evals-on-traces">
    Learn more about configuring evaluation tasks for trace data.
  </Card>
</CardGroup>