Multimodal evaluation is the practice of testing an AI system whose inputs or outputs include images, audio, or video as well as text. It matters because a system can produce a plausible text answer while misreading an image or mishearing audio, and grading only the final text hides that failure.
For example, consider an agent that reads a photographed receipt and decides whether an expense is reimbursable. It correctly reads the total as 84.50 and cites line items that really appear in the image. But it approves a meal that is over the policy cap. Scoring each step separately shows that reading the image worked and applying the policy failed.
A text-only score of the final answer would show a failure with no cause. Multimodal evaluation scores each relevant modality and the task outcome together.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why multimodal evaluation matters
A multimodal model may read a chart, listen to a call, watch a screen, or generate an image. Your evaluation has to follow that data, or it will certify behavior it never checked.
Multimodal evaluation is useful for:
- Locating failures: Tell whether a wrong answer came from perception, reasoning, or the final action.
- Choosing what to fix: Avoid swapping the vision or speech model when the defect is in reasoning over correct inputs.
- Catching modality-specific regressions: See when audio or image quality drops while text quality stays flat.
- Validating judges: Confirm that an automated evaluator can actually read the image or hear the audio it grades.
A strong text-only evaluation score says little about image or audio behavior. Each modality needs its own cases.
How multimodal evaluation works
Multimodal evaluation splits a run into layers and scores each one. Most cases fall into five groups.
- Perception: Did the system extract the right facts from the image or audio?
- Grounding: Did it use the part of the image, or the moment in the audio, that it claims to have used?
- Cross-modal consistency: Does the answer agree with all inputs, or only with the text?
- Generation quality: Is a produced image or spoken reply fit for purpose?
- Task outcome: Did the overall job succeed?
Evaluate the modality-specific step and the end task separately. A correct transcript can still lead to a wrong action. A wrong transcript can also be masked by a lucky answer.
Choosing an evaluator for each layer
Each group calls for a different method. Use mechanical checks wherever ground truth is mechanical.
- Known labels: Perception against a known value can use exact or structured comparison.
- Transcription: Transcription quality can use word error rate.
- Open-ended judgments: Whether a description matches an image, or whether a spoken reply was appropriate, usually needs human review. It can also use a large language model (LLM) judge given the same multimodal input.
An LLM judge is itself a system you need to validate. A judge that is strong on text may be weak at reading charts or hearing tone.
The Arize AX cookbook on evaluating applications shows this layered approach. So does the walkthrough on tracing and evaluating Gemini audio with Arize, which captures the sound and transcript in the trace and attaches evaluations to the layers that need them.
Worked example: a receipt-review agent
Here is the opening example scored layer by layer. The agent receives a receipt image and the expense policy text, then returns approve or decline with a reason.
Case: receipt image + policy text -> approve / decline + reason
Layered scores for one run:
- Perception (code/OCR check): total extracted 84.50, expected 84.50 PASS
- Grounding (judge with image): cited line items exist in image PASS
- Policy application (judge with policy): meal over cap not flagged FAIL
- Final decision matches expected: expected decline, got approve FAIL
Diagnosis: reading the image was fine; applying the cap rule failed.
A text-only score of the final answer would show FAIL with no cause.
Perception uses a code check, labeled here as optical character recognition (OCR), against a known total. Grounding and policy application use judges that see the image and the policy text.
The layering is what makes the result useful. Without it, a team might respond by changing the vision model. The real defect is in policy reasoning over correctly extracted numbers.
How to build and run multimodal evaluations
Start from the trace
Each case should store everything the system saw and produced, as linked spans:
- The original media, or a durable reference to it.
- Any intermediate transcripts or extractions.
- The tool calls.
- The final output.
This lets an evaluator, human or model, see the same evidence the system had. If audio, image, transcript, tool call, and answer are not on one timeline, you cannot attribute a failure to a layer.
Show the judge the real media
LLM evaluation covers judge design and validation in general. Multimodal evaluation adds two requirements.
First, show the judge the image or audio itself rather than a caption of it. Second, check judge accuracy separately for each modality.
Build cases that isolate modalities
Design cases that change one thing at a time:
- Degraded input: The same question with a degraded image.
- Noise: The same audio with background noise added.
- Rewording: The same image with the question reworded.
- Disagreement: Cases where text and image deliberately disagree, so you can see which one the system follows.
Include expected intermediate values where they exist, such as the correct total or the correct transcript of the key sentence. Don’t rely on final answers alone.
Agent-native evaluation extends this thinking to the full run, including steps, tool calls, and final state. That is often where cross-modal errors become visible.
Monitor each modality in production
In production, sample real runs for scoring. Track modality-specific rates over time, such as transcription error on sampled calls, perception failures on sampled images, and downstream task success.
Keep these rates separate in your AI observability views. One merged quality number can stay flat while one modality degrades. Report evaluation metrics per layer, matching the diagnosis structure above.
Multimodal evaluation vs. text evaluation with media attached
A common shortcut is to caption or transcribe the media, then run a text evaluation on the result. That approach evaluates the caption pipeline and the text behavior combined.
It can also certify a system whose judge never saw the image. If the production system reads the image directly, the evaluator must read it too.
The shortcut is acceptable only when production also consumes the caption. In that case, the caption really is the system’s input.
Multimodal evaluation also differs from judging a media-generation feature on looks alone. Whether an image is pleasing and whether it shows the correct product, count, and label are different checks. Task-grounded cases should score the second one explicitly.
Limitations of multimodal evaluation
- Ground truth is expensive: Labeling what an image shows or what a caller said takes longer than labeling text. Ambiguous cases, like a mumbled amount or a partly visible line item, need an adjudication rule, not just more labelers.
- Judges share perception limits: A model judge inherits the limits of its own image or audio handling. It can confidently agree with a wrong extraction that it also failed to see.
- Media raises privacy questions: Audio and video raise storage, retention, and privacy issues more sharply than text traces do. Decide retention before you instrument, not after.
- Layers can pass while the whole fails: The image was read correctly and the policy was quoted correctly, but the wrong line item was compared. Keep end-to-end task cases alongside the layered checks.
Frequently asked questions
Do I need a multimodal judge, or can I evaluate the transcript and final text?
If your system consumes raw media, you need a judge or reviewer that sees the raw media too. Evaluating only derivatives means the judge checks a different input than the system used. Use media-capable judges or human review on at least a sample, and validate whichever judge you use for each modality.
Which metrics should I start with for multimodal evaluation?
Start with the metrics that can be checked mechanically. Use extraction accuracy against known values for perception, word error rate for transcription, and expected final state or decision for task outcome. Add judged criteria only where no mechanical check exists, and validate those judges against human labels.
How many cases does a multimodal evaluation need?
You need enough cases to cover each modality’s known failure patterns, including degraded media, disagreement between modalities, and ambiguous inputs. Keep the unit of analysis at the case or session level. A smaller, well-stratified set beats a large set of clean, easy examples.
Can one aggregate score summarize multimodal quality?
Only as a dashboard headline, never for diagnosis or release decisions. Report results per layer and per modality so a regression in one modality cannot hide inside a stable average.
Putting multimodal evaluation to work
For each multimodal case, record the original media, intermediate extractions, tool calls, final output, and expected values for each layer. Score each layer with the simplest method that works, and validate any judge per modality.
For voice systems specifically, see voice agent evaluation. The Gemini audio walkthrough above shows Arize capturing audio and transcripts in a trace and attaching evaluations to them, which keeps per-layer scores tied to the evidence behind them.
