> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# List evaluator templates

> Retrieve the built-in LLM-as-a-judge evaluator templates. This is the same
catalog the product offers when creating an evaluator, spanning response
quality, code quality, trajectory, RAG, security, and session evals.

Each template carries the judge prompt, the labels it returns, the score
for each label, its optimization direction, and the granularity it
evaluates at.

**Creating an evaluator from a template.** Pick a template, then call
`POST /v2/evaluators` with its fields mapped onto the request:

| Template field | Where it goes in `POST /v2/evaluators` |
| --- | --- |
| `column_name` | `version.template_config.name` |
| `template` | `version.template_config.template` |
| `classification_choices` | `version.template_config.classification_choices` |
| `direction` | `version.template_config.direction` |
| `data_granularity` | `version.template_config.data_granularity`. Send `SPAN`, or omit it, when the template's value is `null` |
| `display_name` | a label for your own use; reuse it for the evaluator's `name` or `description` |
| `rails` | no destination; `classification_choices` already carries the same labels |

Then add the fields a template doesn't carry: `space_id`, `name`, and
`type: TEMPLATE` on the evaluator; a `version.commit_message`; and the
execution settings `template_config.include_explanations`,
`use_function_calling_if_available`, and `llm_config`. Finally, create a
task to run the evaluator.

A complete request built from the `hallucination` template:

```json
{
  "space_id": "U3BhY2U6NDkzOkJaSkc=",
  "name": "hallucination",
  "description": "Built from the hallucination template",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Initial version from built-in template",
    "template_config": {
      "name": "hallucination",
      "template": "You are evaluating whether an answer is factual given reference text...\n{input}\n{output}",
      "classification_choices": { "hallucinated": 1, "factual": 0 },
      "direction": "MINIMIZE",
      "data_granularity": "SPAN",
      "include_explanations": true,
      "use_function_calling_if_available": true,
      "llm_config": {
        "ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
        "model_name": "gpt-4o",
        "invocation_parameters": { "temperature": 0 },
        "provider_parameters": {}
      }
    }
  }
}
```

**Scope:** this returns only the built-in catalog, which is identical for
every caller and contains no space, account, or user data. It does not
include the evaluators that already exist in your space. List those with
`GET /v2/evaluators`.

**Pagination:** not paginated. The catalog is a small fixed list (28
templates, roughly 32 KB of JSON) and the full set is always returned, so
there is no `cursor` or `limit`.

<Warning>This endpoint is in alpha, read more [here](https://arize.com/docs/ax/rest-reference#api-version-stages).</Warning>




## OpenAPI

````yaml https://api.arize.com/v2/spec.yaml get /v2/evaluator-templates
openapi: 3.0.3
info:
  title: Arize REST API
  version: 2.0.0
  description: |
    API specification for the backend data server. The API is hosted globally
    at https://api.arize.com/v2 or in your own environment.
  license:
    name: Apache-2.0
    url: https://www.apache.org/licenses/LICENSE-2.0
servers:
  - description: Global
    url: https://api.arize.com
  - description: Regional
    url: https://api.{region}.arize.com
    variables:
      region:
        default: eu-west-1a
        enum:
          - eu-west-1a
          - ca-central-1a
  - description: Custom Host
    url: https://{host}
    variables:
      host:
        default: api.arize.com
security:
  - bearerAuth: []
tags:
  - name: AI Integrations
    description: |
      AI integrations configure access to external LLM providers (e.g. OpenAI,
      Azure OpenAI, AWS Bedrock, Vertex AI). Integrations can be scoped to the
      entire account, a specific organization, or a specific space.
  - name: Annotation Configs
    description: >
      Annotation configs allow you to define consistent annotation schemas that

      can be reused across your workspace, ensuring evaluations are structured
      and

      comparable over time.
  - name: Annotation Queues
    description: >
      Annotation queues help you organize and manage human evaluation workflows.

      Use queues to assign spans or examples to annotators for review and
      labeling.
  - name: API Keys
    description: >
      API keys are used to authenticate requests to the Arize API. List your
      keys

      to view metadata; the raw secret is never returned after creation.
  - name: Audit Logs
    description: >
      Audit logs record authenticated user actions within an account, providing
      a

      chronological trail for security and compliance review. Access requires

      account admin privileges and audit logging to be enabled.
  - name: Datasets
    description: |
      Datasets are structured, version-controlled example collections you use to
      run, evaluate, and track LLM experiments.
  - name: Evaluators
    description: >
      Evaluators are reusable evaluation configurations used to assess the
      quality

      of LLM outputs. They can be template-based (using LLM judges) or
      code-based.
  - name: Experiments
    description: >
      Experiments let you systematically test prompt/model changes using
      datasets,

      tasks, and evaluators.
  - name: Integrations
    description: >
      Integrations configure access to external LLM providers (e.g. OpenAI,

      Azure OpenAI, AWS Bedrock, Vertex AI), notifications services (e.g.
      PagerDuty, Slack), and

      your own agents. Integrations can be scoped to the entire account, a
      specific

      organization, or a specific space.
  - name: Monitors
    description: >
      Monitors continuously track a metric over your model or LLM application
      data

      and alert you when it crosses a threshold. Each monitor watches a single

      metric - data quality, model performance, drift, a custom metric, or a

      tracing metric - and moves between statuses as the metric passes in and
      out of its 

      healthy range.
  - name: Organizations
    description: >
      Organizations are top-level containers within an Arize AX account for
      grouping spaces.
  - name: Projects
    description: |
      Projects represent LLM applications being monitored in Arize where you can
      observe traces and spans.
  - name: Prompts
    description: >
      Prompts are reusable, versioned templates for LLM interactions. Use
      prompts

      to standardize and manage how you interact with LLMs across your
      application.
  - name: Resource Restrictions
    description: |
      Endpoints for restricting and unrestricting resources (projects, models).
  - name: Role Bindings
    description: |
      Role bindings assign a role to a user on a resource. REST currently
      supports space- and project-scoped bindings.
  - name: Roles
    description: >
      Roles define sets of permissions that can be assigned to users within an

      account. Create custom roles to tailor access control to your team's
      needs.
  - name: Spaces
    description: >
      Spaces are containers within an organization for grouping related
      projects,

      datasets, and experiments, enabling collaboration or isolated
      experimentation

      with role-based access control.
  - name: Spans
    description: |
      Spans represent individual operations within a trace. A span captures the
      timing, status, and attributes of a single operation in your application.
  - name: Tags
    description: |
      Tags are reusable labels defined once per space and attached to resources
      across the platform, so the same vocabulary can be applied to projects,
      datasets, prompts, and more. Tags are shared: renaming or deleting a tag
      affects every resource it is attached to.
  - name: Tasks
    description: |
      Tasks are configurable units of work that tie one or more evaluators to a
      data source (project or dataset). Use tasks to automate evaluation of LLM
      outputs, with support for continuous evaluation and backfill runs.
  - name: Traces
    description: |
      A trace is the collection of spans sharing a trace ID, representing a
      single end-to-end request through an LLM application. Use the Traces
      endpoint to retrieve traces with all of their spans in one call.
  - name: Users
    description: >
      Users represent members of an account. The Users endpoints allow creating,

      listing, updating (display name), and removing users from the account
      programmatically.
  - name: Webhooks
    description: >
      Webhooks are organization-owned destinations that receive event deliveries

      over HTTPS. Deliveries are authenticated with a bearer token or signed
      with

      an HMAC signing secret — the secret is returned exactly once, when the

      webhook is created. Delivery attempts are recorded and can be listed for

      debugging. To choose which events a webhook receives, manage its

      subscriptions through the prompt and evaluator webhook-subscription

      endpoints.
paths:
  /v2/evaluator-templates:
    get:
      tags:
        - Evaluators
      summary: List evaluator templates
      description: >
        Retrieve the built-in LLM-as-a-judge evaluator templates. This is the
        same

        catalog the product offers when creating an evaluator, spanning response

        quality, code quality, trajectory, RAG, security, and session evals.


        Each template carries the judge prompt, the labels it returns, the score

        for each label, its optimization direction, and the granularity it

        evaluates at.


        **Creating an evaluator from a template.** Pick a template, then call

        `POST /v2/evaluators` with its fields mapped onto the request:


        | Template field | Where it goes in `POST /v2/evaluators` |

        | --- | --- |

        | `column_name` | `version.template_config.name` |

        | `template` | `version.template_config.template` |

        | `classification_choices` |
        `version.template_config.classification_choices` |

        | `direction` | `version.template_config.direction` |

        | `data_granularity` | `version.template_config.data_granularity`. Send
        `SPAN`, or omit it, when the template's value is `null` |

        | `display_name` | a label for your own use; reuse it for the
        evaluator's `name` or `description` |

        | `rails` | no destination; `classification_choices` already carries the
        same labels |


        Then add the fields a template doesn't carry: `space_id`, `name`, and

        `type: TEMPLATE` on the evaluator; a `version.commit_message`; and the

        execution settings `template_config.include_explanations`,

        `use_function_calling_if_available`, and `llm_config`. Finally, create a

        task to run the evaluator.


        A complete request built from the `hallucination` template:


        ```json

        {
          "space_id": "U3BhY2U6NDkzOkJaSkc=",
          "name": "hallucination",
          "description": "Built from the hallucination template",
          "type": "TEMPLATE",
          "version": {
            "commit_message": "Initial version from built-in template",
            "template_config": {
              "name": "hallucination",
              "template": "You are evaluating whether an answer is factual given reference text...\n{input}\n{output}",
              "classification_choices": { "hallucinated": 1, "factual": 0 },
              "direction": "MINIMIZE",
              "data_granularity": "SPAN",
              "include_explanations": true,
              "use_function_calling_if_available": true,
              "llm_config": {
                "ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
                "model_name": "gpt-4o",
                "invocation_parameters": { "temperature": 0 },
                "provider_parameters": {}
              }
            }
          }
        }

        ```


        **Scope:** this returns only the built-in catalog, which is identical
        for

        every caller and contains no space, account, or user data. It does not

        include the evaluators that already exist in your space. List those with

        `GET /v2/evaluators`.


        **Pagination:** not paginated. The catalog is a small fixed list (28

        templates, roughly 32 KB of JSON) and the full set is always returned,
        so

        there is no `cursor` or `limit`.


        <Warning>This endpoint is in alpha, read more
        [here](https://arize.com/docs/ax/rest-reference#api-version-stages).</Warning>
      operationId: list_evaluator_templates
      responses:
        '200':
          $ref: '#/components/responses/ListEvaluatorTemplatesResponse'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '403':
          $ref: '#/components/responses/Forbidden'
        '429':
          $ref: '#/components/responses/RateLimitExceeded'
components:
  responses:
    ListEvaluatorTemplatesResponse:
      description: The full list of built-in evaluator templates.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ListEvaluatorTemplatesResponse'
          example:
            evaluator_templates:
              - column_name: hallucination
                display_name: Hallucination
                template: >-
                  You are evaluating whether an answer is factual given
                  reference text.


                  [Input]: {input}

                  [Reference]: {context}

                  [Answer]: {output}


                  Respond with a single word: factual or hallucinated.
                rails:
                  - hallucinated
                  - factual
                classification_choices:
                  hallucinated: 1
                  factual: 0
                direction: MINIMIZE
                data_granularity: null
              - column_name: session_frustration
                display_name: Session Frustration
                template: >-
                  You are given a multi-turn session between a user and an AI
                  assistant.


                  {conversation}


                  Respond with a single word: frustrated or ok.
                rails:
                  - frustrated
                  - ok
                classification_choices:
                  frustrated: 1
                  ok: 0
                direction: MINIMIZE
                data_granularity: SESSION
    Unauthorized:
      description: Authentication is required
      content:
        application/problem+json:
          schema:
            $ref: '#/components/schemas/Problem'
          example:
            status: 401
            title: Authentication required
            detail: You must be authenticated to access this resource.
            instance: /resource
            type: >-
              https://arize.com/docs/ax/rest-reference/errors#authentication-required
    Forbidden:
      description: Insufficient permissions to access this resource
      content:
        application/problem+json:
          schema:
            $ref: '#/components/schemas/Problem'
          example:
            status: 403
            title: Access forbidden
            detail: You do not have permission to access this resource.
            instance: /resource/12345
            type: https://arize.com/docs/ax/rest-reference/errors#access-forbidden
    RateLimitExceeded:
      description: Rate limit exceeded
      headers:
        Retry-After:
          description: |
            When throttled (429), how long to wait before retrying. Value is
            either a delta-seconds integer.
          schema:
            type: integer
            minimum: 0
          example: 42
      content:
        application/problem+json:
          schema:
            $ref: '#/components/schemas/Problem'
          example:
            status: 429
            title: Rate limit exceeded
            detail: >-
              You have exceeded the allowed number of requests. Please try again
              later.
            instance: /resource
            type: >-
              https://arize.com/docs/ax/rest-reference/errors#rate-limit-exceeded
  schemas:
    ListEvaluatorTemplatesResponse:
      type: object
      required:
        - evaluator_templates
      properties:
        evaluator_templates:
          type: array
          items:
            $ref: '#/components/schemas/EvaluatorTemplate'
          description: |
            Every built-in template, ordered by category as the product presents
            them (response quality, code quality, trajectory, RAG, security,
            session).
      additionalProperties: false
      x-forward-compatible: true
    Problem:
      type: object
      description: RFC 9457 Problem Details
      properties:
        title:
          type: string
          description: A short, human-readable summary of the problem type
        status:
          type: integer
          description: >-
            The HTTP status code generated by the origin server for this
            occurrence of the problem
        type:
          type: string
          format: uri-reference
          description: A URI reference that identifies the problem type
        detail:
          type: string
          description: >-
            A human-readable explanation specific to this occurrence of the
            problem
        instance:
          type: string
          format: uri-reference
          description: >-
            A URI reference that identifies the specific occurrence of the
            problem
      required:
        - title
        - status
      additionalProperties: false
      x-forward-compatible: true
    EvaluatorTemplate:
      type: object
      description: >
        A built-in LLM-as-a-judge evaluator template. Templates are the same

        catalog offered in the product's create-evaluator flow, and are
        identical

        for every caller. They carry no space, account, or user data.


        A template is a starting point for an evaluator. To create one from it,
        map

        its fields onto `POST /v2/evaluators`. See the field-by-field mapping
        and a

        complete example on `GET /v2/evaluator-templates`.
      required:
        - column_name
        - display_name
        - template
        - rails
        - classification_choices
        - direction
        - data_granularity
      properties:
        column_name:
          type: string
          description: >
            Stable identifier for the template, and the eval column name it
            writes

            to by default (e.g. `hallucination`). Unique across all templates.
          example: hallucination
        display_name:
          type: string
          description: Human-readable name shown in the product.
          example: Hallucination
        template:
          type: string
          description: >
            The judge prompt. Variables are single-brace, f-string style (e.g.

            `{input}`, `{output}`, `{context}`) and are bound to real data by a

            task's column mappings when the evaluator runs.


            This is the only prompt you need. To have the judge explain its
            label,

            set `include_explanations` on `POST /v2/evaluators`. The explanation

            request is added at run time, not by editing this prompt.
          example: |-
            You are evaluating whether the answer is factual...
            {input}
            {output}
        rails:
          type: array
          items:
            type: string
          description: |
            The labels the judge is allowed to return, in the order the product
            displays them.
          example:
            - hallucinated
            - factual
        classification_choices:
          type: object
          additionalProperties:
            type: number
          description: >
            Maps each label to its numeric score. Pass this through unchanged
            when

            creating an evaluator, since the labels must match those named in
            the

            template.
          example:
            hallucinated: 1
            factual: 0
        direction:
          allOf:
            - $ref: '#/components/schemas/OptimizationDirection'
          description: >
            Whether a higher score is better (`MAXIMIZE`), worse (`MINIMIZE`),
            or

            neither (`NONE`). Controls how trends are rendered. Pass it through

            unchanged, since it must agree with `classification_choices`. If the
            two

            disagree, the product renders the trend backwards.
        data_granularity:
          allOf:
            - $ref: '#/components/schemas/DataGranularity'
          nullable: true
          description: >
            The unit this template evaluates. `null` means span level, which is
            the

            default for most response-quality, RAG, and security templates.

            `SESSION` templates score a whole conversation and require spans
            that

            carry a session identifier.
      additionalProperties: false
      x-forward-compatible: true
    OptimizationDirection:
      type: string
      enum:
        - MAXIMIZE
        - MINIMIZE
        - NONE
      default: NONE
      description: |
        The direction for optimization. Defaults to `NONE` when omitted.
        - MAXIMIZE: higher scores are better
        - MINIMIZE: lower scores are better
        - NONE: higher or lower scores are neither better nor worse
    DataGranularity:
      type: string
      enum:
        - SPAN
        - TRACE
        - SESSION
      description: |
        Data granularity level for evaluation.
        - SPAN - Evaluate at the individual span level.
        - TRACE - Evaluate at the full trace level.
        - SESSION - Evaluate at the session level.
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: <api-key>
      description: >
        Most Arize AI endpoints require authentication. For those endpoints that
        require authentication, include your API key in the request header using
        the format

        ``` Authorization: Bearer <api-key>

        ```

````