Elastic Net Regression: A Method Guide

1. Introduction

1.1 The Problem

Ordinary least squares becomes unreliable when there are many predictors relative to the number of rows, or when predictors are strongly correlated: coefficient estimates swing widely from sample to sample, and with more predictors than rows the solution is not unique. Regularization adds a penalty on the size of the coefficients, accepting a little bias for a lower variance.

1.2 What This Paper Covers

• How ridge, lasso and elastic net differ, and what each does to correlated predictors
• The elastic net objective, how it is fitted, and how its two parameters are chosen
• A reproducible simulation showing the grouping behavior and its cost
• Practical guidance, each point of which can be tested on your own data

2. Background

2.1 Ridge and Lasso

Ridge regression (Hoerl and Kennard, 1970) adds an L2 penalty, the sum of squared coefficients. It shrinks all coefficients toward zero and handles correlated predictors smoothly, but it keeps every predictor in the model. The lasso (Tibshirani, 1996) adds an L1 penalty, the sum of absolute coefficients. It can set coefficients exactly to zero, so it selects variables.

2.2 Limits of Each

Ridge does not produce a sparse model. The lasso, when several predictors are strongly correlated, tends to keep one of them and drop the others, and which one it keeps can change from sample to sample. When there are more predictors than rows, the lasso can keep at most as many predictors as there are rows.

2.3 Elastic Net

Zou and Hastie (2005) proposed combining the two penalties. The L1 part gives sparsity; the L2 part makes the fitted coefficients of strongly correlated predictors closer to each other, which is often called the grouping effect, and removes the lasso's cap on the number of selected predictors.

3. Approach

3.1 The Objective

For responses y (n values) and predictors X (n by p), elastic net minimizes

(1 / 2n) * ||y - X b||^2  +  lambda * [ alpha * ||b||_1  +  (1 - alpha) / 2 * ||b||_2^2 ]

Here lambda >= 0 sets the overall strength of the penalty and alpha, between 0 and 1, sets the mix: alpha = 1 is the lasso and alpha = 0 is ridge regression. Naming differs between tools. In scikit-learn, lambda is called alpha and the mix is called l1_ratio; in R's glmnet, lambda is lambda and the mix is alpha.

3.2 Fitting

The objective is convex. Coordinate descent, which updates one coefficient at a time with a soft-thresholding step, is the standard algorithm (Friedman, Hastie and Tibshirani, 2010). Fitting a decreasing sequence of lambda values and starting each fit from the previous solution (a warm start) is usually much cheaper than fitting each value from scratch.

3.3 Choosing the Two Parameters

• Standardize the predictors first, using the training rows only, because the penalty depends on coefficient size and therefore on predictor scale.
• Choose a grid of lambda values (often log-spaced) and a few values of alpha, and use k-fold cross-validation to estimate out-of-sample error for each pair.
• Pick the pair with the lowest error, or the sparsest pair within one standard error of the lowest (the one-standard-error rule).
• Refit on all training rows with the chosen pair and score a held-out set once.

3.4 What to Report

Report held-out error against a baseline such as ridge or the lasso, the number of nonzero coefficients, and how stable the selected set is across bootstrap refits. Selected coefficients are shrunken, so treat them as ranking and direction evidence, not as unbiased effect sizes, and do not read them as causal.

4. A Reproducible Simulation

The simulation below builds data with three blocks of five predictors. Within a block, predictors share a common component so their pairwise correlation is 0.9. Only the first two blocks affect the outcome (each of those ten predictors has a coefficient of 1); the third block and 20 further columns are noise. Each data set has 100 rows and Gaussian noise with standard deviation 3. Both models are tuned by 5-fold cross-validation.

import numpy as np
from sklearn.linear_model import LassoCV, ElasticNetCV
from sklearn.preprocessing import StandardScaler

def make_data(rng, n=100, group_size=5, n_groups=3, n_noise=20, rho=0.9):
    """Three blocks of correlated predictors plus pure-noise columns.
    Only the first two blocks drive the outcome."""
    blocks = []
    for _ in range(n_groups):
        shared = rng.normal(size=(n, 1))
        blocks.append(np.sqrt(rho) * shared + np.sqrt(1 - rho) * rng.normal(size=(n, group_size)))
    X = np.hstack(blocks + [rng.normal(size=(n, n_noise))])
    beta = np.zeros(X.shape[1])
    beta[:2 * group_size] = 1.0
    y = X @ beta + rng.normal(scale=3.0, size=n)
    return X, y, beta

rows = {"lasso": [], "elastic net": []}
rng = np.random.default_rng(42)
for rep in range(200):
    X, y, beta = make_data(rng)
    Xt, yt, _ = make_data(rng, n=1000)
    sc = StandardScaler().fit(X)
    models = {"lasso": LassoCV(cv=5, random_state=0),
              "elastic net": ElasticNetCV(l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9], cv=5, random_state=0)}
    for name, m in models.items():
        m.fit(sc.transform(X), y)
        sel = m.coef_ != 0
        rows[name].append((sel[:10].sum(), sel[10:].sum(), np.mean((yt - m.predict(sc.transform(Xt))) ** 2)))

for name, r in rows.items():
    r = np.array(r)
    print(f"{name:12s} true predictors kept (of 10): {r[:,0].mean():.2f}   "
          f"noise columns kept (of 25): {r[:,1].mean():.2f}   test MSE: {r[:,2].mean():.2f}")

Output over the 200 simulated data sets:

lasso        true predictors kept (of 10): 8.23   noise columns kept (of 25): 4.25   test MSE: 10.50
elastic net  true predictors kept (of 10): 9.65   noise columns kept (of 25): 6.96   test MSE: 10.30

In this setup elastic net recovered more of the true predictors and also kept more noise columns, with a test error close to the lasso's. The noise variance is 9, so both are near the best test error attainable. These are averages over one simulated design; a different correlation, signal strength or sample size can change the picture.

4.1 Stability Under Resampling

The second block takes one simulated data set (it continues from the previous block and reuses make_data), tunes both models once, then refits each on 200 bootstrap resamples and counts how often each column of the first correlated block is kept.

import numpy as np
from sklearn.linear_model import Lasso, ElasticNet, LassoCV, ElasticNetCV
from sklearn.preprocessing import StandardScaler

# make_data is defined in the previous block
rng = np.random.default_rng(7)
X, y, beta = make_data(rng)
Xs = StandardScaler().fit_transform(X)

lasso = LassoCV(cv=5, random_state=0).fit(Xs, y)
enet = ElasticNetCV(l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9], cv=5, random_state=0).fit(Xs, y)
print(f"chosen: lasso alpha={lasso.alpha_:.3f}; elastic net alpha={enet.alpha_:.3f}, l1_ratio={enet.l1_ratio_}")

n_boot = 200
freq = {"lasso": np.zeros(X.shape[1]), "elastic net": np.zeros(X.shape[1])}
for b in range(n_boot):
    idx = rng.integers(0, len(y), len(y))
    freq["lasso"] += Lasso(alpha=lasso.alpha_).fit(Xs[idx], y[idx]).coef_ != 0
    freq["elastic net"] += ElasticNet(alpha=enet.alpha_, l1_ratio=enet.l1_ratio_).fit(Xs[idx], y[idx]).coef_ != 0

print("share of 200 bootstrap refits in which each column of the first correlated block is kept")
for name, f in freq.items():
    print(f"  {name:12s}", " ".join(f"{v / n_boot:.2f}" for v in f[:5]))

chosen: lasso alpha=0.294; elastic net alpha=0.387, l1_ratio=0.5
share of 200 bootstrap refits in which each column of the first correlated block is kept
  lasso        0.95 0.92 0.73 0.55 0.65
  elastic net  0.99 1.00 1.00 0.97 0.99

The lasso kept some columns of the block much less often than others, which is the instability described in section 2.2. Elastic net kept the whole block almost every time. This is one data set, so it illustrates the behavior and does not measure how often it occurs.

5. Practical Guidance

6. Conclusion

Elastic Net is a sensible choice when predictors are correlated or outnumber the rows and a sparse, steadier model is wanted. In the simulation here it kept correlated predictors together at the cost of keeping more noise columns, with a test error close to the lasso's. Whether that trade is worth it on your data is something to settle with cross-validation, stability checks and a held-out set.