> ## Documentation Index > Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Remote evaluators > Host your own scoring endpoint. You keep the evaluation logic; Arize owns orchestration, retries, and writing results back to traces and datasets. **You keep the scoring logic. Arize owns the orchestration.** A remote evaluator is an HTTP endpoint you host. Your service decides how to score each record — rules, proprietary models, an existing eval platform, whatever you already trust. Arize handles selecting data, calling your endpoint, retries, writing results back onto spans, traces, or sessions, and making those scores usable for filters, charts, and alerts. You never have to rebuild your evaluators inside Arize, and you don't have to build continuous online-eval plumbing yourself. Implement the endpoint in any language. Arize only cares about the HTTP contract. Start with a stub that returns fixed JSON, get Arize talking to it, then plug in your real scoring. ## When to use a remote evaluator | Use a remote evaluator when… | Prefer something else when… | | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | You already have scoring logic or an eval platform you want to keep | You want Arize to run a prompt template for you → [LLM-as-a-judge](/ax/evaluate/create-evaluators#llm-as-a-judge) | | Governance requires scoring to stay on your infrastructure | You need deterministic checks and are fine running Python in Arize → [Code evals](/ax/evaluate/evaluators/code-evaluations) | | Your scoring needs packages or runtimes not supported by [code evals](/ax/evaluate/evaluators/code-evaluations) | You need an agent to explore trace context without column mapping → [Agent as a Judge](/ax/evaluate/agent-as-a-judge) | | You want continuous online evals without building orchestration | | ## Before you start 1. A place to host an HTTPS endpoint Arize can reach on the public internet 2. An Arize space that already has the traces or dataset you want to score 3. (Recommended) A shared secret for auth headers ## The contract ### Request Arize sends ```http theme={null} POST /your-path Content-Type: application/json ``` ```json theme={null} { "metadata": { "request_id": "uuid-for-this-call", "evaluator": "my-relevance-eval", "record_id": "span-or-example-id" }, "input": { "input": "Which restaurants deliver?", "output": "Olio e Piu does." } } ``` | Field | What it is | | --------------------- | ------------------------------------------------------------------------------------------------------------ | | `metadata.request_id` | Correlation id for this call (useful in your logs) | | `metadata.evaluator` | Evaluator name in Arize (handy if one endpoint serves multiple evals) | | `metadata.record_id` | Span or example id being scored | | `input` | The mapped fields for this record. **You choose the keys** in the Arize UI (for example `input` / `output`). | Treat `input` defensively: only read the fields you need, and ignore anything extra. ### Response Arize expects Return HTTP `2xx` with a JSON object. You must include a **`label`**, a **`score`**, or **both**. `explanation` is optional but recommended. ```json theme={null} { "label": "relevant", "score": 0.95, "explanation": "Answer addresses the question." } ``` Valid examples: ```json theme={null} { "label": "pass", "score": 1 } ``` ```json theme={null} { "score": 0.42, "explanation": "Partial match" } ``` ```json theme={null} { "label": "fail" } ``` A `2xx` response with **neither** `label` nor `score` is treated as a failure. ### Status codes | Your status | Arize behavior | | ----------------------------- | ---------------------------------- | | `2xx` with label and/or score | Success — store the result | | `2xx` with neither | Failure for that record | | `401` / `403` | Fatal auth failure — stop retrying | | `429` | Rate limited — back off and retry | | Other `4xx` | Record failure — do not retry | | `5xx` | Transient — safe to retry | ## Path to “it works” Do these in order. Don't add LLM logic until step 3 works end-to-end. Smallest possible server: accept `POST`, read JSON, return a fixed verdict. ```python Python theme={null} @app.post("/evaluate") def evaluate(): _ = (request.get_json(silent=True) or {}).get("input") or {} return {"label": "ok", "score": 1, "explanation": "stub"}, 200 ``` ```javascript Node.js theme={null} app.post("/evaluate", (req, res) => { const _fields = (req.body && req.body.input) || {}; res.status(200).json({ label: "ok", score: 1, explanation: "stub" }); }); ``` ```bash theme={null} curl -i -X POST http://127.0.0.1:5001/evaluate \ -H "Content-Type: application/json" \ -d '{ "metadata": { "request_id": "local-test", "evaluator": "stub", "record_id": "example-1" }, "input": { "input": "Which restaurants deliver?", "output": "Olio e Piu does." } }' ``` You want HTTP `200` and JSON that includes `label` and/or `score`. Deploy so Arize can reach a full HTTPS URL, for example `https://evals.example.com/evaluate`. Arize calls that URL **verbatim** — it does not append paths. If your host sleeps when idle, send one warm-up request before a real eval run so cold starts don't time out. 1. Create a **remote evaluator** in the same space where your data lives. 2. Paste the full **endpoint URL**. 3. Add **auth headers** if your endpoint requires them (for example `Authorization: Bearer `). 4. Map columns or attributes into the `input` keys your endpoint reads (for example question → `input`, answer → `output`). 5. Create an [eval task](/ax/evaluate/run-evals-on-traces) that attaches this evaluator to a project or dataset. 6. Choose scope (span / trace / session), whether to run on historical data and/or continuously, and a sampling rate if continuous. 7. Run the task. Results land as eval columns (for example `eval..label`, `.score`, `.explanation`) on the scored records. See [View eval results](/ax/evaluate/evals-overview). Log when a request arrives and when you return a verdict. Include `metadata.request_id` and `metadata.record_id`. | What you see | Likely cause | | ------------------------------ | -------------------------------------------------- | | No requests | Wrong URL, auth header, space, or task not running | | Requests + `4xx` | Mapping or validation issue in your handler | | Requests + `5xx` | Your scoring path is throwing or timing out | | `2xx` but Arize marks failures | Response missing both `label` and `score` | ## Make the scoring real Once the stub works end-to-end, replace the hardcoded verdict. The HTTP contract stays the same. ### Rule-based (start here) Regex, keyword checks, length limits, JSON schema validation, allowlists — fast, cheap, reproducible. ```python theme={null} import re PATTERN = re.compile(r"\b(deliver|delivery|uber\s*eats)\b", re.I) def score(fields: dict) -> dict: text = str(fields.get("output") or "") ok = bool(PATTERN.search(text)) return { "label": "pass" if ok else "fail", "score": 1 if ok else 0, "explanation": "Mentions delivery." if ok else "No delivery signal.", } ``` ### Call your own LLM from the endpoint If the score needs a model judgment, your endpoint can call OpenAI, Anthropic, or any other provider itself — then return `label` / `score` / `explanation` like any other remote eval. That is different from [LLM-as-a-judge in Arize](/ax/evaluate/create-evaluators#llm-as-a-judge), where Arize runs the judge prompt for you. Here **you** own the model call; Arize still owns orchestration. * Validate required `input` fields first; return `4xx` if they're missing * Instruct the model to return **only** JSON matching the response shape, then parse it defensively * On provider timeouts or rate limits, return `5xx` or `429` so Arize can retry * Prefer returning both `label` and `score` when your rubric has both ## Production checklist ### Auth Protect endpoints that call paid models: ```python theme={null} import os def unauthorized(request) -> bool: expected = f"Bearer {os.environ['EVAL_AUTH_TOKEN']}" return request.headers.get("Authorization", "") != expected ``` Store the secret in an environment variable. Configure the same value as a header on the remote evaluator in Arize. ### Suppress tracing on the eval path If the service hosting the evaluator **also** sends its own traces to Arize, wrap judge calls so evaluator traffic doesn't create noisy nested traces: ```python theme={null} from openinference.instrumentation import suppress_tracing with suppress_tracing(): raw = call_judge(question, answer) ``` Skip this if the evaluator is a standalone service with no tracing of its own. ### Robust input parsing ```python theme={null} def input_fields(body: dict) -> dict: return (body or {}).get("input") or {} ``` * Never assume a mapped field exists * Ignore unknown fields * Prefer failing a single record with `4xx` over crashing the process ## FAQ No. Any stack that can serve HTTPS JSON works. No. Either is enough. Both is fine. `explanation` is optional. Yes. Use `metadata.evaluator` (or separate paths) to branch. On the scored records as eval columns, typically `eval..label` / `.score` / `.explanation`. Trace- and session-level evals use related prefixes. See [View eval results](/ax/evaluate/evals-overview). Template evaluators: Arize owns both the judge prompt and the orchestration. Remote evaluators: **you** own the scoring logic; **Arize** still owns orchestration (who to score, when, retries, write-back). Use remote when you already have scoring logic, governance requirements, proprietary models, or deterministic checks you don't want to reimplement.