Deep SHAP is a fast way to approximate SHAP values for neural networks. In the shap Python library it is shap.DeepExplainer, and the two names refer to the same thing: an attribution method that combines the backpropagation rules from DeepLIFT, short for Deep Learning Important FeaTures, with the Shapley value framing that underpins SHAP, short for SHapley Additive exPlanations. You hand it a model, a background dataset, and an input, and it returns a signed contribution for every input feature or pixel.
The reason it exists is cost. Computing Shapley values by definition means evaluating the model on every subset of features, which is exponential and out of the question for a network with thousands of inputs. Kernel SHAP approximates by sampling perturbations, which means thousands of forward passes per explanation. Deep SHAP takes a different route: it treats the network as a composition of simple parts and propagates attributions backward through it, so one explanation costs roughly a backward pass per background sample instead of thousands of forward passes. It is fast enough to run per prediction, and it is an approximation, not an exact calculation.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Deep SHAP, implemented as
shap.DeepExplainer, approximates SHAP values for neural networks by backpropagating attributions using DeepLIFT style per-node rules. - Attributions are relative to a background dataset, and they sum to
f(x) - E[f(x)], the model output minus the expected output over that background. - The background choice is a modeling decision. It defines what “compared to what” means in every explanation you produce.
- It requires access to the model graph and gradients, so it cannot be used on a hosted language model API.
- It is faster than sampling based attribution and less exact than TreeSHAP, which computes values exactly for tree ensembles.
How it works
DeepLIFT explains a prediction by comparing the activation of every neuron against its activation on a reference input, then distributing the difference in output back through the network to the inputs. It is a single backward pass rather than a search over feature subsets.
The SHAP work showed that per-node attribution rules of this kind can be chosen so the result approximates Shapley values. Deep SHAP applies those rules and then averages over many background samples rather than one reference, which turns a comparison against one arbitrary baseline into an expectation over a distribution. The resulting attributions satisfy the additivity property approximately: they sum to f(x) - E[f(x)], so every explanation is explicitly stated as a departure from the model’s average behavior on your background data.
Two implications follow directly:
- There is no absolute explanation. Change the background and every number changes. A picture explained against a black image and the same picture explained against a sample of the training set produce different attributions, and neither is wrong.
- The values are signed and additive. They roll up into cohort and global summaries the same way other SHAP variants do, which is the progression described in global, cohort, and local model explainability.
Choosing background data
This is where most Deep SHAP results go wrong, and it gets almost no attention.
Draw the background from real data the model was trained on or currently sees. A random sample of 100 rows is a common starting point, and more samples reduce noise at linear cost. Too few and the expectation is unstable, so the same input explained twice gives visibly different attributions.
For images, a set of real images is usually more informative than a single blank baseline, because a blank baseline makes “everything that is not black” look important. For tabular inputs into a network, sampling from the operating distribution keeps the comparison meaningful rather than comparing against a point no real record occupies.
Whatever you choose, write it down next to the explanation. An attribution without its background is not reproducible.
Where it fits among the explainers
Deep SHAP is the neural network entry in the SHAP family. Use it when you control the model, fall back to Kernel SHAP when you can only call the model, and use TreeSHAP when the model is a tree ensemble, where an exact algorithm makes an approximation pointless.
The closest neighbor in the same library is shap.GradientExplainer, which integrates gradients along a path between the background and the input rather than propagating attributions per node. Also approximate, generally smoother and slower, and a reasonable cross-check when a Deep SHAP result looks strange.
Known limits
It is an approximation. The DeepLIFT rules are chosen to approximate Shapley values, and the quality of the approximation depends on the architecture. Nonlinear interactions between inputs are the hard case, and this is exactly where two attribution methods will disagree on the same prediction.
Feature independence is assumed in practice. Correlated inputs can split credit in ways that read as meaningful but are partly a sampling artifact.
Coverage of operations. Unsupported layers raise an error rather than degrading quietly.
Pixel attributions are noisy. Aggregate to regions before drawing conclusions.
A faithful attribution can still be a useless one. If the network learned a shortcut, Deep SHAP reports the shortcut accurately. Faithfulness to the model is not evidence about the world, the caveat running through explainability across the ML lifecycle.
Deep SHAP and language models
Deep SHAP does not run on a hosted language model. It needs the computation graph and gradients, and an API that returns text exposes neither. Open weights models expose gradients, but tokens are not a fixed feature set and a generated sequence is not one number to divide. For debugging LLM applications today, record the prompt, retrieved context, and tool calls, then score the output with an LLM as a judge that emits a rationale.
FAQ
What is the difference between DeepExplainer and GradientExplainer?
Both approximate SHAP values for neural networks against a background dataset. DeepExplainer propagates attributions using DeepLIFT style per-node rules. GradientExplainer uses expected gradients, integrating gradients along paths between background samples and the input. GradientExplainer is usually slower and applies more broadly across architectures; running both is a cheap sanity check.
How many background samples should I use?
Start around 100 drawn from real data and increase until the attributions stop moving between runs. Cost grows roughly linearly with the sample count. If two runs on the same input disagree noticeably, the background is too small.
Is Deep SHAP exact?
No. TreeSHAP is exact for tree ensembles; Deep SHAP is an approximation for neural networks. The additivity property holds by construction relative to the background expectation, but the values themselves are estimates of the Shapley values, not the values themselves.
Which frameworks does Deep SHAP support?
TensorFlow and Keras models, plus PyTorch models, with support limited to the operations the implementation knows how to backpropagate through. Coverage changes across releases, so verify against your pinned version before building a workflow on it.
Can I use Deep SHAP for production monitoring?
Yes, with care about cost and background. It is fast enough to run per prediction, so attributions can be logged alongside predictions and compared across cohorts or over time. Keep the background fixed across windows or the comparison is meaningless.
