Q&A on Retrieved Data

You are given a question, an answer and reference text. You must determine whether the
given answer correctly answers the question based on the reference text.

<data>

<question>
{input}
</question>

<reference>
{reference}
</reference>

<answer>
{output}
</answer>

</data>

Your response must be a single word, either "correct" or "incorrect",
and should not contain any text or characters aside from that word.
"correct" means that the question is correctly and fully answered by the answer.
"incorrect" means that the question is not correctly or only partially answered by the
answer.

from phoenix.evals import ClassificationEvaluator
from phoenix.evals.llm import LLM

QA_TEMPLATE = """You are given a question, an answer and reference text. You must determine whether the
given answer correctly answers the question based on the reference text.

<data>

<question>
{input}
</question>

<reference>
{reference}
</reference>

<answer>
{output}
</answer>

</data>

"correct" means that the question is correctly and fully answered by the answer.
"incorrect" means that the question is not correctly or only partially answered by the answer."""

qa_evaluator = ClassificationEvaluator(
    name="qa_correctness",
    prompt_template=QA_TEMPLATE,
    model=LLM(provider="openai", model="gpt-4o"),
    choices={"incorrect": 0, "correct": 1},
)

result = qa_evaluator.evaluate({
    "input": "What is the capital of France?",
    "reference": "Paris is the capital and largest city of France.",
    "output": "The capital of France is Paris."
})