Remote evaluators

You keep the scoring logic. Arize owns the orchestration. A remote evaluator is an HTTP endpoint you host. Your service decides how to score each record — rules, proprietary models, an existing eval platform, whatever you already trust. Arize handles selecting data, calling your endpoint, retries, writing results back onto spans, traces, or sessions, and making those scores usable for filters, charts, and alerts. You never have to rebuild your evaluators inside Arize, and you don’t have to build continuous online-eval plumbing yourself. Implement the endpoint in any language. Arize only cares about the HTTP contract.

Start with a stub that returns fixed JSON, get Arize talking to it, then plug in your real scoring.

When to use a remote evaluator

Use a remote evaluator when…	Prefer something else when…
You already have scoring logic or an eval platform you want to keep	You want Arize to run a prompt template for you → LLM-as-a-judge
Governance requires scoring to stay on your infrastructure	You need deterministic checks and are fine running Python in Arize → Code evals
Your scoring needs packages or runtimes not supported by code evals	You need an agent to explore trace context without column mapping → Agent as a Judge
You want continuous online evals without building orchestration

Before you start

A place to host an HTTPS endpoint Arize can reach on the public internet
An Arize space that already has the traces or dataset you want to score
(Recommended) A shared secret for auth headers

The contract

Request Arize sends

POST /your-path
Content-Type: application/json

{
  "metadata": {
    "request_id": "uuid-for-this-call",
    "evaluator": "my-relevance-eval",
    "record_id": "span-or-example-id"
  },
  "input": {
    "input": "Which restaurants deliver?",
    "output": "Olio e Piu does."
  }
}

Field	What it is
`metadata.request_id`	Correlation id for this call (useful in your logs)
`metadata.evaluator`	Evaluator name in Arize (handy if one endpoint serves multiple evals)
`metadata.record_id`	Span or example id being scored
`input`	The mapped fields for this record. You choose the keys in the Arize UI (for example `input` / `output`).

Treat input defensively: only read the fields you need, and ignore anything extra.

Response Arize expects

Return HTTP 2xx with a JSON object. You must include a label, a score, or both. explanation is optional but recommended.

{
  "label": "relevant",
  "score": 0.95,
  "explanation": "Answer addresses the question."
}

Valid examples:

{ "label": "pass", "score": 1 }

{ "score": 0.42, "explanation": "Partial match" }

{ "label": "fail" }

A 2xx response with neither label nor score is treated as a failure.

Status codes

Your status	Arize behavior
`2xx` with label and/or score	Success — store the result
`2xx` with neither	Failure for that record
`401` / `403`	Fatal auth failure — stop retrying
`429`	Rate limited — back off and retry
Other `4xx`	Record failure — do not retry
`5xx`	Transient — safe to retry

Path to “it works”

Do these in order. Don’t add LLM logic until step 3 works end-to-end.

Stand up a stub endpoint

Smallest possible server: accept POST, read JSON, return a fixed verdict.

@app.post("/evaluate")
def evaluate():
    _ = (request.get_json(silent=True) or {}).get("input") or {}
    return {"label": "ok", "score": 1, "explanation": "stub"}, 200

app.post("/evaluate", (req, res) => {
  const _fields = (req.body && req.body.input) || {};
  res.status(200).json({ label: "ok", score: 1, explanation: "stub" });
});

Prove the contract with curl

curl -i -X POST http://127.0.0.1:5001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "metadata": {
      "request_id": "local-test",
      "evaluator": "stub",
      "record_id": "example-1"
    },
    "input": {
      "input": "Which restaurants deliver?",
      "output": "Olio e Piu does."
    }
  }'

You want HTTP 200 and JSON that includes label and/or score.

Deploy publicly

Deploy so Arize can reach a full HTTPS URL, for example https://evals.example.com/evaluate.Arize calls that URL verbatim — it does not append paths.

If your host sleeps when idle, send one warm-up request before a real eval run so cold starts don’t time out.

Create a remote evaluator in the same space where your data lives.
Paste the full endpoint URL.
Add auth headers if your endpoint requires them (for example Authorization: Bearer <token>).
Map columns or attributes into the input keys your endpoint reads (for example question → input, answer → output).
Create an eval task that attaches this evaluator to a project or dataset.
Choose scope (span / trace / session), whether to run on historical data and/or continuously, and a sampling rate if continuous.
Run the task.

Results land as eval columns (for example eval.<name>.label, .score, .explanation) on the scored records. See View eval results.

Confirm Arize is calling you

Log when a request arrives and when you return a verdict. Include metadata.request_id and metadata.record_id.

What you see	Likely cause
No requests	Wrong URL, auth header, space, or task not running
Requests + `4xx`	Mapping or validation issue in your handler
Requests + `5xx`	Your scoring path is throwing or timing out
`2xx` but Arize marks failures	Response missing both `label` and `score`

Make the scoring real

Once the stub works end-to-end, replace the hardcoded verdict. The HTTP contract stays the same.

Rule-based (start here)

Regex, keyword checks, length limits, JSON schema validation, allowlists — fast, cheap, reproducible.

import re

PATTERN = re.compile(r"\b(deliver|delivery|uber\s*eats)\b", re.I)

def score(fields: dict) -> dict:
    text = str(fields.get("output") or "")
    ok = bool(PATTERN.search(text))
    return {
        "label": "pass" if ok else "fail",
        "score": 1 if ok else 0,
        "explanation": "Mentions delivery." if ok else "No delivery signal.",
    }

Call your own LLM from the endpoint

If the score needs a model judgment, your endpoint can call OpenAI, Anthropic, or any other provider itself — then return label / score / explanation like any other remote eval. That is different from LLM-as-a-judge in Arize, where Arize runs the judge prompt for you. Here you own the model call; Arize still owns orchestration.

Validate required input fields first; return 4xx if they’re missing
Instruct the model to return only JSON matching the response shape, then parse it defensively
On provider timeouts or rate limits, return 5xx or 429 so Arize can retry
Prefer returning both label and score when your rubric has both

Production checklist

Auth

Protect endpoints that call paid models:

import os

def unauthorized(request) -> bool:
    expected = f"Bearer {os.environ['EVAL_AUTH_TOKEN']}"
    return request.headers.get("Authorization", "") != expected

Store the secret in an environment variable. Configure the same value as a header on the remote evaluator in Arize.

Suppress tracing on the eval path

If the service hosting the evaluator also sends its own traces to Arize, wrap judge calls so evaluator traffic doesn’t create noisy nested traces:

from openinference.instrumentation import suppress_tracing

with suppress_tracing():
    raw = call_judge(question, answer)

Skip this if the evaluator is a standalone service with no tracing of its own.

Robust input parsing

def input_fields(body: dict) -> dict:
    return (body or {}).get("input") or {}

Never assume a mapped field exists
Ignore unknown fields
Prefer failing a single record with 4xx over crashing the process

FAQ

Do I have to use Python?

No. Any stack that can serve HTTPS JSON works.

Do I need both label and score?

No. Either is enough. Both is fine. explanation is optional.

Can one endpoint power multiple evaluators?

Yes. Use metadata.evaluator (or separate paths) to branch.

Where do results show up?

On the scored records as eval columns, typically eval.<evaluator_name>.label / .score / .explanation. Trace- and session-level evals use related prefixes. See View eval results.

What's the difference between this and an LLM template evaluator in Arize?

Template evaluators: Arize owns both the judge prompt and the orchestration.Remote evaluators: you own the scoring logic; Arize still owns orchestration (who to score, when, retries, write-back). Use remote when you already have scoring logic, governance requirements, proprietary models, or deterministic checks you don’t want to reimplement.

Quickstart

Instrument

Observe

Evaluate

Improve

Agents

Settings

Security

When to use a remote evaluator

Before you start

The contract

Request Arize sends

Response Arize expects

Status codes

Path to “it works”

Make the scoring real

Rule-based (start here)

Call your own LLM from the endpoint

Production checklist

Auth

Suppress tracing on the eval path

Robust input parsing

FAQ

​When to use a remote evaluator

​Before you start

​The contract

​Request Arize sends

​Response Arize expects

​Status codes

​Path to “it works”

​Make the scoring real

​Rule-based (start here)

​Call your own LLM from the endpoint

​Production checklist

​Auth

​Suppress tracing on the eval path

​Robust input parsing

​FAQ

When to use a remote evaluator

Before you start

The contract

Request Arize sends

Response Arize expects

Status codes

Path to “it works”

Make the scoring real

Rule-based (start here)

Call your own LLM from the endpoint

Production checklist

Auth

Suppress tracing on the eval path

Robust input parsing

FAQ