AdaBoost: A Comprehensive Technical Analysis

4.1 Estimators, Depth and Learning Rate

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier

# UCI Wisconsin Diagnostic Breast Cancer data, bundled with scikit-learn: 569 samples, 30 features, two classes
X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

def cv_accuracy(model):
    scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
    return scores.mean(), scores.std()

print(f"{len(y)} samples, {X.shape[1]} features, {y.mean():.1%} positive class\n")

print("Number of estimators (stumps, learning rate 1.0):")
for n in (10, 50, 100, 200, 500):
    m, s = cv_accuracy(AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=n, random_state=0))
    print(f"  {n:>4} estimators: accuracy {m:.3f} (fold sd {s:.3f})")

print("\nBase learner depth (100 estimators):")
for depth in (1, 2, 3, 5):
    m, s = cv_accuracy(AdaBoostClassifier(DecisionTreeClassifier(max_depth=depth), n_estimators=100, random_state=0))
    print(f"  depth {depth}: accuracy {m:.3f} (fold sd {s:.3f})")

print("\nLearning rate (stumps, 100 estimators):")
for lr in (1.0, 0.5, 0.1):
    m, s = cv_accuracy(AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, learning_rate=lr, random_state=0))
    print(f"  learning rate {lr}: accuracy {m:.3f} (fold sd {s:.3f})")

Output

569 samples, 30 features, 62.7% positive class

Number of estimators (stumps, learning rate 1.0):
    10 estimators: accuracy 0.947 (fold sd 0.011)
    50 estimators: accuracy 0.974 (fold sd 0.011)
   100 estimators: accuracy 0.972 (fold sd 0.007)
   200 estimators: accuracy 0.975 (fold sd 0.012)
   500 estimators: accuracy 0.977 (fold sd 0.007)

Base learner depth (100 estimators):
  depth 1: accuracy 0.972 (fold sd 0.007)
  depth 2: accuracy 0.979 (fold sd 0.007)
  depth 3: accuracy 0.970 (fold sd 0.009)
  depth 5: accuracy 0.965 (fold sd 0.016)

Learning rate (stumps, 100 estimators):
  learning rate 1.0: accuracy 0.972 (fold sd 0.007)
  learning rate 0.5: accuracy 0.967 (fold sd 0.007)
  learning rate 0.1: accuracy 0.963 (fold sd 0.014)

Estimators. Accuracy rose from 0.947 with 10 estimators to 0.974 with 50, then moved between 0.972 and 0.977 up to 500. After about 50 estimators the differences are within one fold spread, so extra estimators cost training time without a clear gain here.

Depth. Depth 2 scored 0.979 against 0.972 for stumps, a gap about the size of the fold spread (0.007). Depth 3 and 5 were no better (0.970 and 0.965, the latter with a larger spread of 0.016).

Learning rate. With stumps and 100 estimators, the rate of 1.0 scored 0.972, 0.5 scored 0.967 and 0.1 scored 0.963. Smaller rates did not help on this data at this number of estimators; they would need more estimators to catch up.

4.2 Sensitivity to Label Noise

Here a share of the training labels is flipped at random and the model is scored on untouched test folds, for AdaBoost with stumps and for a random forest.

# Label noise: flip a share of the TRAINING labels, score on untouched test folds
def noisy_cv(make_model, noise, seed=1):
    rng = np.random.default_rng(seed)
    accs = []
    for train, test in cv.split(X, y):
        y_train = y[train].copy()
        flip = rng.random(len(y_train)) < noise
        y_train[flip] = 1 - y_train[flip]
        accs.append(make_model().fit(X[train], y_train).score(X[test], y[test]))
    return np.mean(accs)

rows = []
for noise in (0.0, 0.1, 0.2, 0.3):
    rows.append({"label noise": f"{noise:.0%}",
                 "AdaBoost (stumps, 100)": noisy_cv(lambda: AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, random_state=0), noise),
                 "Random forest (100)": noisy_cv(lambda: RandomForestClassifier(n_estimators=100, random_state=0), noise)})
print(pd.DataFrame(rows).round(3).to_string(index=False))

Output

label noise  AdaBoost (stumps, 100)  Random forest (100)
         0%                   0.972                0.965
        10%                   0.933                0.951
        20%                   0.902                0.928
        30%                   0.863                0.896

Both methods lose accuracy as noise grows, but AdaBoost loses more: at 10% noise it drops 3.9 points (0.972 to 0.933) against 1.4 points for the random forest (0.965 to 0.951), and at 30% noise it scores 0.863 against 0.896. This is the practical meaning of the sensitivity described in section 2.3. It does not say AdaBoost is a poor choice, only that label quality matters more for it.

Recommendation 1: Check the Labels First

Because AdaBoost is sensitive to wrong labels, review a sample of the labels before training and fix or remove obvious errors. Unusual feature values (outliers) are a separate matter; an outlier detector such as Isolation Forest helps with those, not with wrong labels.

Recommendation 2: Start From the Library Defaults and Tune by Cross-Validation

Begin with the library defaults (scikit-learn uses 50 estimators and a learning rate of 1.0) and stumps, then vary the number of estimators, then the learning rate, and only then the depth of the base learner. Stop adding estimators when accuracy under cross-validation stops improving. The fold spread in section 4.1 is a rough guide to noise, not a test, so compare paired differences on identical folds before treating a gain as real.

Recommendation 3: Handle Class Imbalance Explicitly and Measure the Rare Class

For unbalanced data, try class weights, resampling and threshold adjustment, and judge them on recall and precision for the rare class. A simple class weight is n_samples / (n_classes * np.bincount(y)), passed through the sample weights.

Recommendation 4: Monitor After Deployment

Track accuracy (and rare-class recall, if relevant) on fresh labelled data, and compare the distribution of the input features with the training data, for example with a Kolmogorov-Smirnov test per feature. Decide in advance what degradation triggers retraining.

Recommendation 5: Compare With Alternatives Under the Same Protocol

Evaluate AdaBoost against a random forest and a gradient boosting model with identical preprocessing and the same cross-validation folds, and record accuracy, training time and the interpretability you need.