Evaluators

Definition and types of Evaluators. Score abstraction.

At the core, an Evaluator is anything that returns a Score. Evaluators can be split into two broad categories:

• LLM-based: evaluators that use an LLM to perform the judgement.

  • Examples: hallucination, document relevance

• Heuristic: evaluators that use a deterministic process or calculation.

  • Examples: exact match, BLEU, precision

Scores

• Every score has the following properties:

  • name: The human-readable name of the score/evaluator.

  • source: The origin of the evaluation signal (llm, heuristic, or human)

  • direction: The optimization direction; whether a high score is better or worse

• Scores may also have some of the following properties:

  • score: numeric score

  • label: The categorical outcome (e.g., "good", "bad", or other label).

  • explanation: A brief rationale or justification for the result.

  • metadata: Arbitrary extra context such as model details, intermediate scores, or run info.

Properties of Evaluators

All phoenix-evals Evaluators have the following properties:

• Sync and async evaluate methods for evaluating a single record or example

• Single record evals return a list of Score objects. Oftentimes, this is a list of length 1 (e.g. exact_match), but some evaluators return multiple scores (e.g. precision-recall).

• A discoverable input_schema that describes what inputs it requires to run.

• Evaluators accept an arbitrary eval_input payload, and an optional input_mapping which map/transforms the input to the shape they require.