Evaluate Receipt Agents with an Image Judge

Runnable Gradio app

Run the companion app with 14 fictional receipt images and URL-based trace inputs.

Overview

In this tutorial, you will trace a receipt-extraction agent that takes an image of a receipt, and produces structured JSON with details. You will then create a span-level LLM-as-a-judge evaluator that decides whether the extracted JSON is supported by the image. This tutorial assumes you have basic knowledge of:

tracing

LLM-as-a-judge evaluators

By the end, you will be able to:

Trace an image reference, extraction output, and image metadata together.
Map a receipt image and extracted JSON to a visual-groundedness evaluator.
Distinguish supported outputs, unsupported assertions, and images that require human review.

Why visual groundedness needs its own evaluator

A JSON-shaped response from a receipt-extraction agent can be valid while still inventing a total, merchant, or line item. Text-only correctness checks cannot see the evidence. A visual-groundedness evaluator receives both the receipt image and the extraction, so it can ask one focused question: does the output say only what the image supports? The companion app uses gpt-5.4-mini to extract receipt data. The tested evaluator configuration uses gpt-5.6-luna, a stronger image-aware model, to judge it. If your AX AI integration exposes a different judge model, use that model consistently in the evaluator and app metadata.

Before you start

You will need:

An OpenAI API key with access to gpt-5.4-mini for extraction and an AX AI provider integration with gpt-5.6-luna for judging.
Python 3.10 or later.
The example receipt app from the Arize tutorials repo

Run the receipt intake app

Clone Arize-ai/tutorials, then install and configure the companion app:

cd tutorials/python/cookbooks/receipt_image_evals
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Copy the .env.example file to .env, then set the relevant API keys and Arize AX space Id. Then run the app:

python receipt_app.py

You will see a local URL printed to the terminal. Open this to see the receipt parser.

❯ python receipt_app.py

* Running on local URL:  http://127.0.0.1:7860

From the app, select some receipts, then click Process expense to process the image. From Arize AX, open the receipt-image-evals project, and look at the trace.

A trace for a processed receipt with the image in the input

Create the image-aware judge

Create a categorical LLM-as-a-judge evaluator for visual groundedness.

Create and name the evaluator

In the left sidebar, select Evaluators. Click New Evaluator, select LLM-as-a-Judge, then choose Create from blank. Name it Receipt Visual Groundedness.

Set the scope to span

Set the evaluator scope to Span. Each receipt.extract span contains one self-contained receipt image input and its extracted JSON output, so the judge does not need the rest of the trace or session.

Add the template

For Judge Model, select your OpenAI provider integration and gpt-5.6-luna. Then use the following template. It gives the judge the visual-grounding rubric and separates the image-bearing input from the JSON it must assess. You will map receipt_image and extraction to span fields in a later step.

Assess whether the structured extraction is visually grounded in the receipt
image.

<receipt_image>
{receipt_image}
</receipt_image>

<extraction>
{extraction}
</extraction>

Choose grounded only when every asserted merchant, item, amount, currency,
and total is visibly supported. Choose not_grounded for invented or
contradicted values. Choose needs_review when the image is degraded or
ambiguous enough that a reliable decision cannot be made.

Select the classification label using the configured choices. In the AX
evaluator's `explanation` field, return only valid JSON, without Markdown
fences or any other text, with exactly these fields:

{
  "evaluated_extraction": <the extraction JSON exactly as received>,
  "judge_result": {
    "label": <the selected classification label>,
    "reason": <a concise visual-grounding rationale>
  }
}

Add the choices

Add these three classification choices and scores. Set the optimization direction to maximize, so the aggregate score represents the share of receipts with safely grounded extractions.

Label	Score	Meaning
`grounded`	`1`	The merchant, line items, values, and review decision are supported by the receipt image.
`not_grounded`	`0`	The result asserts content that the image does not support.
`needs_review`	`0`	The receipt is too degraded or ambiguous to evaluate with confidence.

Configure the data source and matching spans

In Configuration, set Data Source to the receipt-image-evals project. Under Filter Spans, filter to name = "receipt.extract" so the evaluator receives only the extraction spans that contain the receipt image and structured output. Confirm that Matching Spans shows processed receipt.extract spans before continuing.

Map variables to data

Under Map Variables to Data, map receipt_image to attributes.input.value and extraction to attributes.output.value. Check the preview for each mapping: the first should contain the receipt image URL and the second the extracted JSON.

Run continuously and backfill

Leave Run Continuously enabled so new matching extraction spans are evaluated as they arrive. Turn on One-Time Backfill to also evaluate the processed receipt spans that already exist in the project.

Create and run the evaluator

Click Create and Run Evaluator to save the evaluator and start the backfill and continuous evaluation task.

Review the eval results

Now you have the evaluator running, you can review the evaluator results for the existing receipts that have been processed, and for any new traces. Select a trace in the project, then select its receipt.extract span. In the Evaluations section, open Receipt Visual Groundedness to see the label, score, and explanation. The explanation contains the JSON extraction evaluated by the judge and its visual-grounding review.

A receipt extraction trace showing the visual-groundedness evaluation result

Summary

In this tutorial, you:

Traced receipt-image extraction with an image URL and structured JSON output.
Created a span-level LLM judge to assess whether each extraction is visually grounded.
Configured continuous evaluation and a backfill, then reviewed the label, score, and JSON explanation on the extraction span.

Next steps

Align with human judgment

Calibrate the judge against reviewed examples before relying on it for production decisions.

Run online evals

Set sampling, filters, and monitoring for evaluators running on live traces.

Human review

Send ambiguous receipt results to a review queue and use the labels to improve the evaluator.

Instrument

Observe

Evaluate

Improve

Advanced Workflows

Evaluate Receipt Agents with an Image Judge

Runnable Gradio app

Overview

Why visual groundedness needs its own evaluator

Before you start

Run the receipt intake app

Create the image-aware judge

Review the eval results

Summary

Next steps

Align with human judgment

Run online evals

Human review

Runnable Gradio app

​Overview

​Why visual groundedness needs its own evaluator

​Before you start

​Run the receipt intake app

​Create the image-aware judge

​Review the eval results

​Summary

​Next steps

Align with human judgment

Run online evals

Human review

Overview

Why visual groundedness needs its own evaluator

Before you start

Run the receipt intake app

Create the image-aware judge

Review the eval results

Summary

Next steps