Error analysis is the process of grouping failures into categories so a team knows what to fix. Instead of treating every bad output as a one-off, you look for patterns: retrieval miss, stale context, judge misalignment, prompt ambiguity, tool outage, policy gap.
It is the least glamorous work in AI engineering and it is where most eval quality comes from. A dashboard can only report categories somebody already thought to define. Error analysis is how the categories get invented, which means a team that skips it monitors metrics describing last quarter’s understanding of its own failures.
The output is not a document. It is a ranked list of failure categories with counts attached, and each category that survives becomes a check, a dataset slice, a prompt change, or a product decision.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Error analysis produces your eval suite. You cannot write a useful judge for a category you have not read 20 examples of.
- Sample deliberately, not uniformly. Stratify by low scores, negative feedback, escalations, tool errors, and high step counts, and still read some random sessions.
- Label the first thing that went wrong, not the symptom the user reported. Symptom-named categories cannot be acted on.
- Rank by frequency times cost. The category that is 4% of failures and produces wrong refunds outranks the 30% that is cosmetic.
- An
otherbucket above roughly a fifth of your labels means the taxonomy is wrong, not that your failures are unusual.
The loop
1. Pull a sample you can actually read
Start with 30 to 50 failed or suspect sessions. That is enough to see the shape of the distribution and small enough to finish in an afternoon, which matters more than statistical elegance because an analysis nobody completes has no value.
Where to pull from, in rough order of yield: negative user feedback, sessions that escalated to a human, the lowest eval scores, sessions containing tool errors, sessions with unusual step counts or cost, and then a plain random sample of successful-looking traffic. That last one is not optional. Filtered sampling only surfaces failures your filters already recognize, and the failure that reports success is invisible to every filter you have.
2. Read the whole trajectory and write one line
For each session, read it in order and write one sentence describing what went wrong first. Not “wrong answer.” Something like “called search_orders with the customer name instead of the order ID and then answered from an empty result set.”
Resist categorizing while you read. Open-ended notes first, categories second. Start with a taxonomy and you will fit every session into it, including the ones that do not belong, and the interesting failure is usually the one that does not fit.
3. Cluster the notes into categories
Now group the sentences. Good categories are causal, exclusive enough that most sessions land in exactly one, and specific enough to imply a fix. “Retrieval returned nothing and the model answered anyway” is a category. “Quality issue” is not.
Published failure taxonomies are useful here as a starting vocabulary rather than as an answer. A field analysis of how production agents break will name categories you have not thought of, and your own pile will contain two or three that are specific to your tools and appear on no generic list.
4. Count, and weight by cost
Frequency alone points you at whatever is noisiest. Multiply it by a cost your product already measures: escalation rate, refund exposure, retries, abandoned sessions, tokens burned per failed session. The ranking usually reorders once cost is included, and the reorder is the useful part.
Report counts with the denominator and the sampling method attached. A category representing 22% of a stratified sample of failures is not 22% of traffic, and that distinction gets lost the moment the number reaches a slide.
5. Turn categories into checks
Each stable category becomes something automated:
- Deterministic checks for anything assertable: argument validity, empty result handling, missing required tool call, forbidden phrase, schema violation.
- A judge for the ones needing interpretation. The labeled examples you just produced are the calibration set, which is what separates a useful judge from a plausible prompt, and aligning against human labels is why LLM as a judge works at all.
- A dataset slice so the category is measured separately rather than averaged into a score that hides it.
Then resample later. New categories appear as traffic and tools change, and the ones you fixed should shrink. If a category’s share does not move after the fix ships, the fix addressed something else.
Why aggregate metrics do not replace this
A single score compresses every failure mode into one number, and averaging is the operation that hides a small catastrophic category behind a large benign one. That compression is why aggregate benchmark scores stopped being enough and trace-level analysis is what teams moved to. Two systems scoring 0.84 can fail in completely different ways, and only one may be acceptable to ship.
A metric also cannot tell you why. A drop in correctness is a prompt to go read sessions, and the reading is the analysis.
Common ways this goes wrong
- Labeling the symptom. Categories named after what the user saw produce no fix. Name the first wrong step.
- Overlapping categories. If most sessions get two labels, the taxonomy describes stages rather than causes. Collapse or split until one label fits.
- The junk drawer. A large
otherbucket means the real categories are still hiding inside it. - Reading only the outputs. Without the trajectory you cannot tell a retrieval miss from a model ignoring good context, and both get pooled into one useless category.
- Stopping after one round. This is a cadence, not a project. Once per release cycle is a reasonable default.
FAQ
How is error analysis different from evaluation?
Evaluation scores behavior against criteria you have already chosen. Error analysis discovers what those criteria should be. They run in a cycle: analysis produces categories, categories become evals, evals surface a population of failures, and reading that population is the next round of analysis.
How many traces do I need to read?
Read until new sessions stop producing new categories. In practice that tends to happen faster than people expect, often within a few dozen sessions for a narrow workflow, and much later for an agent with many tools and a wide task surface. Volume matters less than reading each one completely.
Can an LLM do the error analysis for me?
It helps with the mechanical parts: summarizing long trajectories, proposing candidate clusters from your notes, and applying an established label at scale. It should not define the taxonomy, because the categories that matter depend on what your product considers expensive. The usual sequence is humans read and label a sample, then a judge scales the labels once agreement is verified.
Who should do it?
Whoever can decide what counts as a failure. That is usually an engineer and a domain expert together, reading the same sessions. Handing it to a labeling function loses the domain judgment; handing it to a domain expert alone loses the trace-level detail needed to name causes.
Is this the same as error analysis in traditional machine learning?
Same intent, different artifact. Classical error analysis starts from aggregate error on a labeled test set and drills into residuals and confusion between classes. Agent error analysis starts from individual trajectories, often with no ground truth label, and the failure is a decision in a sequence rather than a numeric miss. Slicing by segment and studying the worst slice carries over unchanged.
