Which Variables Actually Matter? With The Uncertainty Attached

How it works

A linear model is fitted for a numeric outcome and a logistic model for a two-level one, and scored by R-squared or AUC on cross-fitted predictions so that no row is ever scored by a model that trained on it. Each predictor is then shuffled inside the held-out folds and the already-fitted model is re-scored: the fall in performance is its permutation importance. The entire procedure (resample the rows, redraw the folds, refit the models, reshuffle) is repeated many times, and the spread across those repeats becomes the interval on each importance and the basis for judging whether one predictor genuinely outranks the next. Alongside it the analysis refits without each predictor (drop-column importance), refits with each predictor alone (standalone performance), reports the direction of each effect, and computes variance inflation plus correlated groups so that overlapping predictors are flagged rather than ranked against one another.

Use it when you have an outcome and several candidate explanations and need to know which columns a model actually depends on, how confident that ranking is, and which columns are duplicating each other.

Not for establishing that changing a predictor would change the outcome. That needs an experiment or a causal design. Not for per-observation explanations, which are a different quantity from the global importance reported here. Not when the real relationship is a strong interaction or a sharp non-linearity, which a linear or logistic model will understate.

What data do you need?

One row per observation, with the outcome and every candidate predictor as columns. For example, weekly marketing performance:

Minimum 60 rows · Best with 300-4,000 rows with 3-12 candidate predictors

What's in the report?

Standard-library analysis: of the columns you think explain an outcome, which ones does a model actually depend on? Each candidate predictor is shuffled in turn and the model is re-scored on rows it never trained on, so the fall in performance measures how much the fit really leans on that column. The whole procedure (split, refit, shuffle) is repeated on resampled data, so every importance arrives with an uncertainty interval instead of a single number, and the report says which steps in the ranking are genuinely separated and which are a coin flip. A drop-column importance, the direction of each effect, a collinearity diagnostic that names correlated groups, and the distinction between redundant and irrelevant come with it. Works on any dataset: map a numeric or yes/no outcome and two or more candidate predictors.