Create evaluator

Creates a new evaluator with an initial version.

Payload Requirements

• The evaluator name must be unique within the given space.
• type (top-level) selects the evaluator kind: TEMPLATE, CODE, REMOTE, or DECISION_JUDGE. With TEMPLATE, provide version.template_config. With CODE, provide version.code_config — where code_config.type is MANAGED or CUSTOM (a separate discriminator within code_config, independent of the top-level type: CODE). With REMOTE, provide version.remote_config.integration_id referencing an accessible EVALUATOR integration. Remote evaluator creation requires the remote evaluators feature to be enabled; otherwise the request returns 403. With DECISION_JUDGE, provide version.decision_judge_config. Decision Judge must be enabled for the account; otherwise the request returns 403. An ai_integration_id that does not exist or is not available in the space returns 404 (LLM integration not found or not accessible to this space). A model_name that is not a Decision Judge model, or questions that break the question rules, return 422. The evaluator runs on spans only. Each question writes its own result, named eval.<evaluator name> <question name>.label and .score.
• For template evaluators: version.template_config.name is the eval column name; must match ^[a-zA-Z0-9_\s\-&()]+$.
• For template evaluators: version.template_config.template is the prompt template; use {variable} for placeholders (f-string format, e.g. {input}, {output}).
• For template evaluators: version.template_config.classification_choices is required and maps choice labels to numeric scores (e.g. {"relevant": 1, "irrelevant": 0}).
• For code evaluators: see CodeConfig — managed evaluators (code_config.type: MANAGED) use managed_evaluator and variables; custom evaluators (code_config.type: CUSTOM) use code, optional imports, and variables.
• System-managed fields (id, created_at, updated_at, created_by_user_id) are rejected on input.

Valid example (template evaluator)

{
  "name": "Hallucination Detector",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Initial version",
    "template_config": {
      "name": "hallucination",
      "template": "Given the input: {input}\nand the output: {output}\nIs the output a hallucination?",
      "include_explanations": true,
      "use_function_calling": true,
      "classification_choices": {"hallucinated": 0, "factual": 1},
      "llm_config": {
        "ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
        "model_name": "gpt-4o",
        "invocation_parameters": {"temperature": 0},
        "provider_parameters": {}
      }
    }
  }
}

Invalid example (type/config mismatch — TEMPLATE type with code_config)

{
  "name": "Bad Evaluator",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Wrong config",
    "code_config": {
      "type": "CUSTOM",
      "name": "my_eval",
      "code": "class Evaluator: ...",
      "variables": ["input"]
    }
  }
}