Explainability is the ability to give a human-readable account of why a model produced a particular output, and which inputs moved it. It covers two questions that get confused constantly: what this model relies on in general, and why it decided one case the way it did. The first is a property of the model, the second a property of a single prediction, and the second is what a customer, an auditor, or an on-call engineer usually asks about.
This is not philosophical. If a fraud model declines a transaction, someone has to say which signals drove it. If a lending decision is contested, a regulator expects specific reasons rather than an average.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Explainability answers “why this output.” Interpretability describes a model already legible without extra tooling.
- Model access decides which method you can run, scope decides whether it answers the question asked, and audience decides the format.
- Attribution methods disagree, and correlated inputs split credit in ways that read as meaningful but are not.
- An explanation describes the model, not the world: not evidence of a causal effect, not proof of fairness.
- Token-level attribution for LLMs is unsettled research. Tracing prompts, retrieved context, and tool calls works today.
Explainability and interpretability are not the same thing
An interpretable model is legible by construction. Explainability is what you do when it is not: a post-hoc method that returns an approximate account of behavior.
Choosing a method: access, scope, audience
Almost every method answers one of two questions. Attribution asks how much each input moved one prediction, covering the SHAP family, gradient methods, local surrogates such as LIME, and counterfactuals. Importance asks how much the model relies on each input across a population, covering permutation, gain based tree scores, and global surrogates. Both rank the same model differently without either being buggy.
Which you run is settled by three questions. Take them in order, because the first eliminates the most options.
What access do you have to the model? Access, not preference, decides the algorithm.
- A tree ensemble you can load. Use TreeSHAP.
- A neural network you control. Use Deep SHAP.
- Any model you can call but not open. Use Kernel SHAP and budget for sampling cost.
- Prediction logs only. Fit a surrogate and quote fidelity.
What scope answers the question that was actually asked? Most explanations that satisfy nobody were computed at the wrong scope rather than computed wrong.
- Global describes what the model relies on across a population: the scope for design review, feature pruning, and catching a feature that dominates more than it should.
- Cohort restricts that question to a slice such as one region, one channel, or one time window. Most real findings live here, because a model can behave well overall and depend on something absurd inside a segment that is 4% of traffic and most of the complaints.
- Local explains one prediction, which is what an adverse action notice, a support escalation, or a single bad agent run requires.
Who reads the output? An engineer wants local attribution next to the trace. A risk reader wants per-decision reasons plus method and reference window. An end user wants a counterfactual, the smallest change that would have flipped the decision.
What explainability does not give you
It does not give you causality. Attribution reports what the model used, and the model learned correlations from training data, so a feature can dominate an explanation and have no causal effect on the outcome. Correlated inputs are also why methods disagree: when two features carry the same information, how credit splits between them is an artifact of the algorithm, not a fact about the model. It does not settle fairness either: a clean report on a model trained on biased history describes the bias precisely and excuses nothing. And it does not survive distribution shift quietly, since every explanation is computed against a reference distribution. That is why it belongs next to monitoring and evaluation rather than in a one-time model card, as argued in what an agent observability platform is.
Explainability for LLMs and agents
This is the honest part. For tabular models, “which feature moved this prediction” has a well-posed answer. For a large language model it does not: tokens have no fixed feature set, the output is a sequence rather than a scalar, hosted models expose no gradients, and token-level attributions are hard to act on.
What works today is a different kind of explanation: record the whole path. A trace shows the prompt sent, the documents retrieved, each tool call, and the final output, using spans, traces, and sessions as the underlying units. Pair it with evals that emit a rationale, as an LLM as a judge does.
A model’s stated reasoning is generated output. It can be wrong about the process that produced the answer, so treat it as a claim to verify, not a log.
FAQ
Do I need explainability if my model is accurate?
Accuracy tells you the model was right on data you already have labels for. It does not tell you what the model will do on a record type you have never scored, and it will not produce a reason when someone contests a decision. The two most common triggers for adding explainability are a regulator or customer asking “why this outcome,” and an accuracy drop where you need to know whether the model changed or an input broke.
What do I need to log now to explain a decision later?
The inputs as scored, the prediction, the model version, and the reference window. Without those four, a dispute six months from now cannot be reconstructed by any method. For an agent, the equivalent record is the run itself.
Does an explanation prove a model is fair?
No. Attribution reveals what the model relies on, including proxies for protected attributes, but any fairness conclusion requires outcome measurement across groups plus a definition of fairness that fits the decision being made.
How often should explanations be recomputed?
On the cadence of the data, not the model. A ranking from the training notebook describes traffic that no longer exists, so recompute on a recent window, hold the reference fixed while comparing windows against each other, and record which method and window produced any explanation you ship. Agent systems need the same discipline applied to traces and evals.
