Feature importance is a score assigned to each input feature that reflects how much that feature contributed to a model’s predictions or to its accuracy. Rank the scores and you get an ordering: this feature carried the model, these five did almost nothing, that one is suspiciously dominant.
The phrase covers a family of methods rather than a single calculation, and the methods do not agree with each other. Some measure how much a feature reduced error while the model was being trained. Some measure how much performance degrades when you corrupt the feature at inference time. Some measure how much a feature moved one individual prediction. Those are three different questions, and “feature importance” gets used for all three, which is where most of the confusion around the term comes from.
Two distinctions keep the rest of this straight. Global importance summarizes behavior across a dataset, while local importance, sometimes called attribution, explains a single prediction. And importance to the model is not the same as effect in the world.
Key takeaways
- Feature importance is a family of methods, not one metric, and different methods legitimately produce different rankings of the same model.
- Impurity or gain based importance is free but biased toward high-cardinality and continuous features.
- Permutation importance measures what the model needs at inference time, and it breaks down when features are correlated.
- SHAP attributions are additive and local, so they roll up to global and cohort views without changing definitions.
- Importance is not causality and usually not signed. Magnitude tells you the feature mattered, not which direction it pushed.
How importance is actually computed
Impurity or gain based. Tree ensembles record how much each split reduced loss, then sum that over every split on a feature. This is what scikit-learn exposes as feature_importances_. It is free, since training already computed it, and it is biased: features with many distinct values get more chances to produce a good-looking split. XGBoost makes the ambiguity visible by offering gain, cover, and weight, which routinely disagree about the same model. A tree importance quoted without saying which one is not interpretable.
Permutation importance. Shuffle one column in held-out data, rescore, and measure how far a metric drops. This asks a cleaner question: how much does the trained model depend on this input at prediction time. Two catches. It needs a metric and labeled holdout data, so the answer changes if you switch from AUC to log loss. And shuffling creates input combinations that never occur, which is how each half of a correlated pair can look unimportant.
Drop column. Remove the feature, retrain, compare. The most direct answer to “do I need this feature,” and the most expensive, at one full retrain per feature.
Shapley value based. SHAP, short for SHapley Additive exPlanations, divides each prediction among its features using the Shapley value from cooperative game theory. The attributions sum to the prediction minus a base value, so a global ranking is just the mean absolute SHAP value per feature, and local and global importance share one definition instead of being two unrelated numbers. Which algorithm produces those values is a function of the model class, exact for tree ensembles via TreeSHAP and approximate elsewhere.
Global, cohort, and local views
A single global ranking is the least useful artifact you can produce, because it averages over every case you care about. The interesting version compares rankings across slices: approved versus declined, one region versus another, last week versus this week. A feature that ranks 14th overall and 2nd inside the segment generating your complaints is the finding, and that progression from population to slice to record is laid out in global, cohort, and local model explainability.
Where importance rankings mislead
Correlation splits credit. Give a model two features that carry nearly the same information and it will use them interchangeably. Every method then has to divide the credit somehow, and the split is an artifact of the algorithm, not a fact about the data. Retrain with a different seed and the ranking can flip while accuracy holds.
Magnitude is not direction. Most importance scores are unsigned. Knowing that income matters does not tell you whether more income raised or lowered the score, let alone the shape of the relationship. For that you need signed local attributions plotted against feature value.
Importance is relative to a distribution. Every score is computed against some reference data. When production drifts, rankings move without anyone touching the model. That is a signal, and it is a reason to recompute importance on recent traffic rather than citing a number from the training notebook.
A dominant feature is often a bug. If one input explains nearly everything, check for leakage before celebrating. Fields written after the outcome is known, IDs that encode the label, and timestamps tied to a labeling process all show up first as impossibly strong features.
What it is used for in production
Feature importance earns its keep in triage. Drift monitoring will tell you a feature’s distribution moved; importance tells you whether that feature matters enough to explain a performance drop, so you can rank a page of drift alerts by expected impact. It also drives feature pruning and bias review, where the question is whether the model leans on a proxy for a protected attribute, and explainability across the ML lifecycle walks through where each check belongs.
Feature attribution for LLMs and agents
Feature attribution does not port cleanly to language models, because a prompt has no fixed set of named features to rank and the output is a sequence rather than one score. What teams do instead is redefine the unit, and that reframing is worth spelling out because it is where the concept on this page survives.
Stop asking which token mattered and pick a unit you can name: the retrieved document, the tool call, the prompt section, the routing decision. Each is a discrete thing the system either used or did not, which makes the usual importance questions answerable again. Which retrieved documents show up in runs that scored well versus runs that failed. Whether removing a prompt section changes the outcome, which is drop column importance with a section in place of a feature. The unit is coarser than a feature, and in exchange the answer is directly actionable: drop a retriever, rewrite a section, remove a tool. It takes enough runs to beat anecdote, which is why this work sits next to tracing and evaluating AI agents.
Plenty of tabular modeling also survives inside agent systems, in rankers, routers, and fraud checks, where ordinary feature importance applies unchanged.
FAQ
What is the difference between feature importance and SHAP values?
SHAP values are one way to compute importance, and they are local by construction: every prediction gets its own signed attribution per feature. Averaging their absolute values gives a global importance ranking. Older tree importances are global only and unsigned, so they cannot explain an individual case.
Why do two methods rank my features differently?
Because they measure different things. Gain based importance describes what helped during training. Permutation importance describes what the trained model depends on at inference time on a specific metric. SHAP describes how credit for each prediction divides among inputs. Disagreement is expected, and correlated features widen it.
Does high feature importance mean the feature causes the outcome?
No. Importance describes the model, and the model learned correlations in its training data. Establishing a causal effect requires an intervention or a causal design, not an attribution report.
How do correlated features affect feature importance?
They dilute and destabilize it. Two features carrying the same signal can each look weak under permutation, because the model compensates using the other one, and Shapley methods that assume feature independence can assign credit to a feature the model barely uses. Group correlated features and interpret the group, or drop duplicates before ranking.
Can I use feature importance for feature selection?
As a first pass, yes, with validation. Rank, cut the tail, retrain, and confirm the metric held. Do not trust a single ranking from a single seed, and prefer permutation or drop column methods here, since they measure predictive contribution rather than training-time split behavior.
