Standard Calibration
Executive Summary

Executive Summary

Whether Predicted Default Probability means what it says about Defaulted

Observations
4000
Calibration
clearly miscalibrated
ECE
0.0805
Calibration Slope
0.587
Brier Score
0.2006
AUC (Discrimination)
0.775
Across 4,000 rows, Predicted Default Probability is clearly miscalibrated as a probability for 'Yes' in Defaulted. The expected calibration error is 0.0805 — the row-weighted average gap between what was predicted and what happened, across 10 bins. On average the model predicted 49.8% against an observed 49.6%, so overall it is neither high nor low by any meaningful margin. A logistic recalibration of the outcome on the predicted logit returns an intercept of -0.011 (0 is ideal) and a slope of 0.587 (1 is ideal): the slope of 0.587 sits entirely below 1, so the predictions are too extreme (over-confident): high predictions run above what actually happens and low predictions run below it. The worst bin that clears the row threshold covers predictions from 0.10 to 0.20: 449 rows predicted an average of 14.7% and 27.8% actually were 'Yes' (95% interval 23.9% to 32.2%). The Brier score of 0.2006 breaks into reliability 0.0077, resolution 0.0561 and uncertainty 0.2500: the reliability term is the only piece a recalibration can remove, and it is 3.8% of the total. The verdict survives re-binning: across 4 different binnings the expected calibration error ranged over just 0.0018 and every one of them reads as clearly miscalibrated. The direction of the error is binning-independent by construction, since it is the average prediction minus the observed base rate; only the size depends on how the bins are cut. Calibration and discrimination are different questions and this report only answers the first. Discrimination asks whether the score puts the right rows at the top; here the AUC is 0.775 (fair). Calibration asks whether the NUMBER is honest. The two come apart in both directions: a model that predicted the base rate of 49.6% for every row would be perfectly calibrated and rank nothing (AUC 0.500), while a score that ranks perfectly can still be numerically wrong everywhere. For the ranking question, run the ROC analysis (standard_roc_analysis) on the same two columns.
Suggested Interpretation

The model is clearly miscalibrated with an expected calibration error of 0.0805. It predicted an average of 49.8% against an observed 49.6%, so it is neither systematically high nor low overall, but its calibration slope of 0.587 (95% CI: 0.544 to 0.630) reveals over-confidence: predictions are too extreme. The worst bin holding 40+ rows (0.10 to 0.20, n=449) predicted 14.7% but saw 27.8% default (95% CI: 23.9% to 32.2%). The Brier score of 0.2006 is only 3.8% reliability — the part recalibration could remove — leaving 96.2% as resolution and uncertainty. The verdict is robust: across four different binnings, expected calibration error ranged over just 0.0018, all reading as clearly miscalibrated. Calibration and discrimination are separate: the model's AUC of 0.775 means it ranks fairly but assigns dishonest numbers.

Overview

Analysis Overview

Calibration of Predicted Default Probability against Defaulted across 4,000 observations.

N Observations4000
Ece0.0805
Brier0.2006
Base Rate0.4965
Suggested Interpretation

When the model says 70% risk, the event does not happen 70% of the time — it is miscalibrated. Across 4,000 observations, the model's predicted probabilities for 'Yes' in Defaulted sit 0.0805 points away from reality on average (the expected calibration error). The root cause is over-confidence: a logistic recalibration returns a slope of 0.587, well below the ideal 1.0, meaning high predictions run above what actually happens and low predictions run below it. The Brier score of 0.2006 breaks down into reliability (0.0077), resolution (0.0561), and uncertainty (0.2500); only the reliability term — just 3.8% of the total — could be fixed by recalibration. The model does know something (it ranks fairly, with AUC 0.775), but the numbers it assigns are not honest.

Data Preparation

Data Quality

Positive-class definition, the probability scale, dropped rows, and class balance.

Initial Rows4000
Final Rows4000
Rows Removed0
Base Rate0.4965
Suggested Interpretation

All 4,000 rows carried both an outcome and a predicted probability with no missing values. The positive class 'Yes' in Defaulted matched a conventional label against 'No'; the split was 1,986 'Yes' (49.6%) and 2,014 'No'. Predicted Default Probability values ranged from 0.001 to 1.000 and were read as probabilities as given, requiring no rescaling. No prediction sat at exactly 0 or 1, so the logit recalibration used the predicted values without adjustment. Rows were grouped into 10 equal-width bins, all of which held at least 40 rows, so no single thin bin drives the picture.

Visualization

Reliability Curve

Observed outcome rate against predicted probability, per bin of Predicted Default Probability.

Suggested Interpretation

The reliability curve shows 5 bins above the perfect-calibration diagonal and 5 below, a roughly even split masking a systematic tilt: the model under-forecasts in the lower half (0.05 to 0.40) and over-forecasts in the upper half (0.60 to 1.00). The worst bin that clears the row threshold covers predictions from 0.10 to 0.20 with 449 rows predicting an average of 14.7% while 27.8% actually defaulted (95% CI: 23.9% to 32.2%), a gap of −0.1317. All 10 bins hold at least 40 rows, so no thin-bin noise is driving the pattern. The tilt is consistent with the slope of 0.587: the model's probability range is compressed, so it undershoots low-risk and overshoots high-risk cases.

Data Table

Bin-by-Bin Calibration

Predicted versus observed per bin, with a binomial confidence interval on every observed rate.

BinNMean PredictedObserved RateCI LowCI HighGapReading
0.00 to 0.105860.04840.1280.10330.1575-0.0796predicted lower than the outcome interval allows (under-forecast here)
0.10 to 0.204490.14670.27840.2390.3216-0.1317predicted lower than the outcome interval allows (under-forecast here)
0.20 to 0.303290.24810.3040.25670.3557-0.0559predicted lower than the outcome interval allows (under-forecast here)
0.30 to 0.403390.3510.42180.37040.475-0.0709predicted lower than the outcome interval allows (under-forecast here)
0.40 to 0.503220.45180.48140.42730.5358-0.0296the prediction falls inside the outcome's confidence interval
0.50 to 0.602790.54930.54120.48260.59870.0081the prediction falls inside the outcome's confidence interval
0.60 to 0.703250.64990.59690.54280.64880.053predicted higher than the outcome interval allows (over-forecast here)
0.70 to 0.803730.75020.64610.59630.69290.1041predicted higher than the outcome interval allows (over-forecast here)
0.80 to 0.904200.8520.74050.69650.78010.1116predicted higher than the outcome interval allows (over-forecast here)
0.90 to 1.005780.9520.84950.8180.87630.1025predicted higher than the outcome interval allows (over-forecast here)
Suggested Interpretation

In the 10 bins, the model shows evidence of miscalibration in 8 of them — that is, where the average prediction falls outside the 95% Wilson confidence interval around the observed rate. The decisive bin is 0.10 to 0.20 with n=449: it predicted 14.7% but observed 27.8% (CI: 23.9% to 32.2%), a gap of −0.1317. This is the largest gap in any bin holding 40+ rows. Only 2 bins give no evidence of miscalibration: 0.40 to 0.50 (predicted 45.2%, observed 48.1%, CI: 42.7% to 53.6%) and 0.50 to 0.60 (predicted 54.9%, observed 54.1%, CI: 48.3% to 59.9%). The pattern shows the model under-forecasting in bins 0.00–0.40 and over-forecasting in bins 0.60–1.00, consistent with over-confidence.

Data Table

Calibration Fit & Error

Calibration intercept and slope from a logistic recalibration, plus expected and maximum calibration error.

TermEstimateCI LowCI HighInterpretation
Calibration intercept (with slope free)-0.0108-0.08140.0597Intercept of the logistic fit of the outcome on the predicted logit. 0 is ideal.
Calibration slope0.58720.54410.6303Slope of that fit. 1 is ideal; below 1 means the predictions are too extreme, above 1 means they are too timid.
Calibration-in-the-large-0.0113-0.0920.0693Intercept when the slope is held at 1 — whether the predictions are too high or too low overall. 0 is ideal.
Mean predicted probability0.4982——The average probability the model put on 'Yes'.
Observed base rate0.4965——The share of rows that actually were 'Yes'.
Expected calibration error (ECE)0.0805——Row-weighted average gap between predicted and observed across the 10 bins used. 0 is perfect.
Maximum calibration error (bins with enough rows)0.1317——The largest gap in any bin holding at least 40 rows — the worst error that is not just small-sample noise.
Maximum calibration error (all bins)0.1317——The largest gap in any non-empty bin, including thin ones. Read it against the row counts in the bin table before acting on it.
Suggested Interpretation

The logistic recalibration of the outcome on the predicted logit returns a slope of 0.587 (95% CI: 0.544 to 0.630), sitting entirely below 1.0 — the model is over-confident. The intercept is −0.0108 (95% CI: −0.0814 to 0.0597), not significantly different from 0, so there is no systematic high-or-low bias overall; the mean predicted probability was 0.4982 against an observed base rate of 0.4965. The expected calibration error is 0.0805 (the row-weighted average gap across bins) and the maximum calibration error is 0.1317 in the bin holding 40+ rows. Neither calibration-in-the-large nor the overall prediction bias drives the problem — the slope below 1 is the signature fault.

Data Table

Brier Score Decomposition

The Brier score split into reliability, resolution and uncertainty.

ComponentValueInterpretation
Brier score0.2006Mean squared distance between the predicted probability and the 0/1 outcome across 4,000 rows. Lower is better; 0 is perfect.
Reliability (lower is better)0.0077How far the predictions sit from what actually happened inside each bin. This is the part of the score that recalibration can remove.
Resolution (higher is better)0.0561How far the bins' outcome rates move away from the overall base rate of 49.6%. This is the part that comes from the model actually knowing something.
Uncertainty (fixed by the data)0.25The variance of the outcome itself, base rate times one minus base rate. No model can reduce this — it is the difficulty of the problem.
Within-bin residual-0.001Brier minus (reliability - resolution + uncertainty). The decomposition is computed over 10 bins, so this leftover is the spread of the predictions inside each bin.
Suggested Interpretation

The short answer

The Brier score of 0.2006 is dominated by uncertainty (0.25), the inherent difficulty of the problem. Reliability—the part that recalibration could fix—is only 0.0077, or 3.8% of the total. This means the miscalibration is real but small relative to the problem's difficulty.

The detail

The Brier score is 0.2006. Reliability is 0.0077, the distance from honesty and the only part a recalibration could remove. Resolution is 0.0561, the credit for pushing bins away from the base rate of 49.6%. Uncertainty is 0.25, the variance of the outcome itself (base rate × (1 − base rate)), fixed by the data and beyond any model's reach. The within-bin residual is −0.001, the spread of predictions inside each of the 10 bins. The decomposition is Reliability − Resolution + Uncertainty = 0.0077 − 0.0561 + 0.25 = 0.2016, which rounds to the reported Brier score of 0.2006 within the residual.

What this can't tell you

A low reliability term does not mean the model is acceptable—it means the miscalibration, while real, is small. The resolution term (0.0561) shows the model does have discriminative power; the low reliability just says it states that power incorrectly on the probability scale.

Visualization

Distribution of Predicted Probabilities

How Predicted Default Probability is spread across the 0 to 1 range.

Suggested Interpretation

The short answer

The model concentrates predictions at the extremes: 29.1% of predictions fall below 0.10 or above 0.90. The busiest bucket is 0.95–1.00 with 8.0% of the 4,000 rows. Only 49.2% of predictions live in the middle range (0.20–0.80), where calibration errors have the most weight.

The detail

The distribution is bimodal, with mass at both ends. The 0.00–0.05 bucket holds 311 rows (0.0778 share), and the 0.95–1.00 bucket holds 320 rows (0.08 share). Together, predictions below 0.10 account for 586 rows (0.1467 share) and above 0.90 account for 1,164 rows (0.291 share). The middle range (0.20–0.80) contains 1,968 rows (0.492 share). Because every calibration number above is a weighted average over this distribution, regions the model rarely visits contribute almost nothing to the headline error however wrong the model is there.

What this can't tell you

The histogram shows where the model makes its strongest claims but does not assess whether those claims are correct. See the reliability curve and bin table for the accuracy of predictions in each range.

Data Table

Does the Verdict Survive Re-binning?

Expected and maximum calibration error recomputed under several binnings.

BinningN BinsEceMceVerdict
5 equal-width bins50.07940.1063clearly miscalibrated
10 equal-width bins100.08050.1317clearly miscalibrated
20 equal-width bins200.08050.1383clearly miscalibrated
10 equal-count bins100.07870.1366clearly miscalibrated
Suggested Interpretation

The short answer

The verdict is robust. Across four different binning schemes, the expected calibration error ranged over only 0.0018 (from 0.0787 to 0.0805), and every scheme reads as clearly miscalibrated. The direction of miscalibration is binning-independent.

The detail

The headline uses 10 equal-width bins with an expected calibration error of 0.0805. Under 5 equal-width bins, the ECE is 0.0794. Under 20 equal-width bins, 0.0805. Under 10 equal-count bins (each holding the same number of rows), 0.0787. The maximum calibration error ranges from 0.1063 (5 bins) to 0.1383 (20 bins), but the expected calibration error stays tight. The direction of the error is binning-independent by construction, since it is the average prediction minus the observed base rate; only the size depends on how the bins are cut. All four verdicts read as clearly miscalibrated.

What this can't tell you

Coarse bins average opposite errors away and read too kind; fine bins split into noise and read too harsh. The range of 0.0018 across these four schemes is small enough to support the headline, but it does not guarantee the verdict would hold under all possible binnings.

Data Table

What Correction Would Buy

Platt scaling and isotonic regression applied to the same predictions, reported in-sample.

MethodBrierEceNote
As delivered (no correction)0.20060.0805The predicted probabilities exactly as supplied.
Platt scaling (logistic recalibration)0.1930.0127The fitted intercept and slope applied back to the predicted logit. It can shift and stretch the probabilities but cannot change their ORDER, so it leaves discrimination untouched. Fitted on these same rows.
Isotonic regression (pool-adjacent-violators)0.19120A non-decreasing step function fitted by pooling adjacent violators, here 36 distinct steps. It is far more flexible than Platt scaling and is therefore the most optimistic in-sample.
Suggested Interpretation

The short answer

A simple logistic recalibration (Platt scaling) would lower the Brier score from 0.2006 to 0.193, and a more flexible isotonic regression would lower it to 0.1912—an in-sample improvement of 4.7%. Neither would improve discrimination (AUC stays at 0.775) because they cannot reorder the rows.

The detail

Platt scaling fits a logistic model of the outcome on the predicted logit and applies it back, using two parameters. It lowered the Brier score to 0.193 and the ECE to 0.0127, in-sample. Isotonic regression fits a non-decreasing step function by pooling adjacent violators, here 36 distinct steps. It lowered the Brier score to 0.1912 and the ECE to 0, in-sample. Because a monotone correction never reorders the rows, neither improves the AUC of 0.775. Both were fitted on the very rows they are scored on, so the improvements shown are in-sample and optimistic—an upper bound on what a correction would deliver, not an estimate of it. This is most severe for the isotonic fit, which is free to bend to this data at 36 separate steps.

What this can't tell you

These in-sample numbers are best-case. Before trusting either correction, fit it on one split of the data and measure it on another. The isotonic fit is especially prone to overfitting and will likely deliver far less than 4.7% improvement on held-out data.

Data Table

Calibration Is Not Discrimination

The ranking question alongside the honesty question, and why they are not the same.

MetricValueInterpretation
AUC (discrimination)0.775Probability that a random 'Yes' row is scored above a random 'No' row — fair ranking. 0.500 means no ranking ability at all.
Observed base rate49.6%The share of rows that were 'Yes'.
Brier score0.20057The delivered probabilities scored against the outcomes.
Brier score of a base-rate-only forecast0.24999What you would score by predicting 49.6% for every single row. That forecast is perfectly calibrated and ranks nothing.
Brier skill score0.1977How much the delivered probabilities beat that base-rate-only forecast; 0 means no better, 1 would be perfect.
Suggested Interpretation

Calibration and discrimination are separate questions. The model has fair discrimination (AUC 0.775, meaning a random 'Yes' row scores above a random 'No' row with probability 0.775) but dishonest numbers. A forecast that simply predicted 49.6% for every row would be perfectly calibrated (reliability = 0) and rank nothing (AUC 0.500). The delivered probabilities score a Brier of 0.20057 against 0.24999 for that base-rate-only forecast, a Brier skill score of 0.1977. All that skill comes from resolution (0.0561), the model's ability to separate the outcome groups; fixing calibration moves the numbers onto the diagonal but cannot make the model know more than it does. For the ranking question, run a standard ROC analysis on the same two columns.

Methodology

Methodology

Statistical methodology and diagnostics for Classifier Calibration

Statistical Method

Classifier Calibration

Standard-library analysis: are the predicted probabilities honest? Map a predicted probability and the true yes/no outcome and get the reliability curve (predicted versus observed rate per bin, against the perfect-calibration diagonal, with a binomial confidence interval on every bin), the Brier score decomposed into reliability, resolution and uncertainty, the calibration intercept and slope from a logistic recalibration, expected and maximum calibration error, the distribution of the predicted probabilities, a check that the verdict survives a different binning, and what Platt scaling and isotonic regression would buy. Answers the question 'when this model says 70%, does it happen 70% of the time?'

Data
N = 4000 observations
Assumptions
  • The mapped score is a predicted probability of the positive outcome, on a 0-to-1 scale (a 0-to-100 percentage is detected and rescaled)
  • Each row is one independent prediction paired with the outcome it was predicting
  • The outcome is genuinely binary; if it has more than two values the analysis becomes one class against the rest
  • Higher predicted probability means more likely to be the positive class
Limitations
  • Calibration is not discrimination — a perfectly calibrated model can be useless, so the ranking question needs the ROC analysis instead
  • The expected and maximum calibration error depend on how the bins are cut; the report recomputes them under several binnings and states whether the verdict held, but the size of the error is never a single exact number
  • Bins holding few rows carry wide confidence intervals and should not drive the narrative; the headline maximum error is taken only from bins that clear a row threshold
  • The Platt and isotonic corrections are fitted and scored on the same rows, so their reported improvement is in-sample and optimistic — most severely for isotonic regression
Software & Citation
MCP Analytics · mcpanalytics.ai
Code Appendix

Analysis Code

Complete R source code for this analysis

Classifier Calibration — Are the Predicted Probabilities Honest?

Takes predicted probabilities plus the true binary outcome and answers the question "when this model says 70%, does it happen 70% of the time?".

Delivers the reliability curve (predicted vs observed rate per bin, against the perfect-calibration diagonal, with binomial confidence intervals per bin), the Brier score decomposed into reliability / resolution / uncertainty, the calibration intercept and slope from a logistic recalibration, expected and maximum calibration error, the histogram of predicted probabilities, a binning-sensitivity check, and the effect of two remedies — Platt scaling and isotonic regression.

Why This Method?

A single number cannot say whether probabilities are honest. The Brier score mixes three different things together; splitting it into reliability (how far the probabilities sit from what happened), resolution (how much they separate outcomes) and uncertainty (how hard the problem was) is what turns one number into a diagnosis. The reliability curve then shows exactly WHERE the model is wrong, and the calibration slope says whether the predictions are systematically too extreme or too timid.

What This Analysis Covers

  • The reliability curve with binomial (Wilson) confidence intervals per bin
  • The Brier score decomposed into reliability, resolution and uncertainty
  • Calibration intercept and slope from a logistic recalibration
  • Expected (ECE) and maximum (MCE) calibration error
  • The distribution of predicted probabilities
  • Whether the verdict survives a different binning
  • Platt scaling and isotonic regression as remedies, reported in-sample
  • Discrimination (AUC) alongside calibration, and why they are not the same

Standard Library

Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {actual, score} — the same input shape as the ROC analysis. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Step 1: Row accounting + semantic column discovery

initial_rows <- nrow(df)
  if (!"actual" %in% names(df)) {
    stop("column_mapping must map an &#x27;actual' column (the true yes/no outcome).")
  }
  if (!"score" %in% names(df)) {
    stop("column_mapping must map a &#x27;score' column (the predicted probability).")
  }
  actual_name <- humanize_semantic("actual", col_map)
  score_name  <- humanize_semantic("score", col_map)

Step 2: Choose the positive class and binarize (no row dropping yet)

v_raw <- df$actual
  if (is.logical(v_raw)) {
    y_all <- as.integer(v_raw)
    positive_label <- "TRUE"
    negative_label <- "FALSE"
    positive_rule  <- "the boolean TRUE value was taken as the positive class"
  } else {
    vc <- trimws(as.character(v_raw))
    vc[is.na(v_raw) | vc == "" | toupper(vc) %in% c("NA", "N/A", "NULL")] <- NA_character_
    lv <- sort(unique(vc[!is.na(vc)]))
    if (length(lv) < 2) {
      stop(sprintf(
        paste0("The outcome column(&#x27;%s') holds only one distinct value (%s) — a ",
               "calibration analysis needs rows where the event happened and rows ",
               "where it did not."),
        actual_name,
        if (length(lv) == 1) paste0("&#x27;", lv[1], "'") else "none"))
    }
    pat <- paste0("^(1|yes|y|true|t|positive|pos|disease|churn|churned|fraud|",
                  "default|defaulted|converted|won|win|success|event|flagged)$")
    hits <- lv[grepl(pat, tolower(lv))]
    if (length(hits) >= 1) {
      positive_label <- hits[length(hits)]
      positive_rule  <- paste0("&#x27;", positive_label,
                               "&#x27; matched a conventional positive label")
    } else {
      positive_label <- lv[length(lv)]
      positive_rule  <- paste0("no conventional positive label was found, so &#x27;",
                               positive_label,
                               "&#x27; (alphabetically last) was used as the positive class")
    }
    neg_levels <- setdiff(lv, positive_label)
    negative_label <- if (length(neg_levels) == 1) neg_levels[1]
                      else paste0("not ", positive_label)
    if (length(lv) > 2) {
      positive_rule <- paste0(positive_rule,
        "; the outcome had more than two distinct values, so this is a one-against-the-rest evaluation")
    }
    y_all <- ifelse(is.na(vc), NA_integer_, as.integer(vc == positive_label))
  }

Step 3: Coerce the score to numeric under the 95% rule

sc_raw <- df$score
  if (is.numeric(sc_raw)) {
    sc_all <- as.numeric(sc_raw)
  } else {
    chr <- trimws(as.character(sc_raw))
    chr[chr == ""] <- NA_character_
    n_nonblank <- sum(!is.na(chr))
    sc_all <- suppressWarnings(as.numeric(chr))
    n_ok <- sum(!is.na(sc_all))
    if (n_nonblank == 0 || n_ok < 0.95 * n_nonblank) {
      stop(sprintf(
        paste0("The predicted-probability column(&#x27;%s') is not numeric — only %s of %s ",
               "non-blank values could be read as numbers, and at least 95%% is required. ",
               "Calibration needs a predicted probability per row, not a text label."),
        score_name, format(n_ok, big.mark = ","), format(n_nonblank, big.mark = ",")))
    }
  }

Step 4: Keep only complete (outcome, probability) pairs

keep <- !is.na(sc_all) & !is.na(y_all)
  p <- as.numeric(sc_all[keep])
  y <- as.integer(y_all[keep])
  final_rows   <- length(y)
  rows_removed <- initial_rows - final_rows

  if (final_rows < 30) {
    stop(sprintf(
      paste0("Only %s rows have both a usable outcome(&#x27;%s') and a numeric predicted ",
             "probability(&#x27;%s') — a calibration analysis needs at least 30."),
      format(final_rows, big.mark = ","), actual_name, score_name))
  }

Step 5: Decide what scale the predictions are on

rng_lo <- min(p); rng_hi <- max(p)
  if (rng_lo >= 0 && rng_hi <= 1) {
    scale_rule <- paste0("the values ran from ", fnum(rng_lo, 3), " to ",
                         fnum(rng_hi, 3), ", so they were read as probabilities as given")
  } else if (rng_lo >= 0 && rng_hi <= 100) {
    scale_rule <- paste0("the values ran from ", fnum(rng_lo, 2), " to ",
                         fnum(rng_hi, 2), ", so they were read as percentages and divided by 100")
    p <- p / 100
  } else {
    stop(sprintf(
      paste0("The column mapped as the predicted probability(&#x27;%s') ranges from %s to %s, ",
             "which is neither a probability(0 to 1) nor a percentage(0 to 100). ",
             "Calibration asks whether predicted PROBABILITIES are honest, so a raw ",
             "score or a logit has no calibration to check. To measure how well such a ",
             "score ranks the two classes of &#x27;%s', use the ROC analysis instead."),
      score_name, fnum(rng_lo, 3), fnum(rng_hi, 3), actual_name))
  }

Step 6: Class-balance guard, named in the user's own columns

n1 <- sum(y == 1L); n0 <- sum(y == 0L)
  if (n1 < 5 || n0 < 5) {
    stop(sprintf(
      paste0("A calibration analysis needs at least 5 rows of each outcome, but among the ",
             "usable rows &#x27;%s' has %s '%s' and %s '%s'."),
      actual_name, format(n1, big.mark = ","), positive_label,
      format(n0, big.mark = ","), negative_label))
  }

  base_rate <- mean(y)
  mean_pred <- mean(p)
  cal_large <- mean_pred - base_rate

Step 7: Brier score and its Murphy decomposition over the main bins

brier <- mean((p - y)^2)
  unc   <- base_rate * (1 - base_rate)
  bss   <- if (unc > 0) 1 - brier / unc else NA_real_

Bin count scales with the data: never fewer than 4, never more than 10, and never more than one bin per 10 rows.

k_main <- max(4L, min(10L, as.integer(floor(final_rows / 10))))
  bins_main <- make_bins(p, y, seq(0, 1, length.out = k_main + 1L))

  reliable_min <- max(10L, as.integer(ceiling(0.01 * final_rows)))
  em <- bin_errors(bins_main, final_rows, reliable_min)
  ece <- em$ece; mce_all <- em$mce_all; mce_reliable <- em$mce_reliable
  n_bins_used <- em$n_bins; thin_bins <- em$thin_bins
  worst_row <- em$worst_row

  rel <- sum(bins_main$n * (bins_main$mean_predicted - bins_main$observed_rate)^2,
             na.rm = TRUE) / final_rows
  res <- sum(bins_main$n * (bins_main$observed_rate - base_rate)^2,
             na.rm = TRUE) / final_rows
  resid <- brier - (rel - res + unc)

Step 8: Logistic recalibration — calibration intercept and slope

The predictions are clamped away from 0 and 1 so the logit is finite.

eps <- min(0.5 / final_rows, 1e-6)
  pc  <- pmin(pmax(p, eps), 1 - eps)
  n_clamped <- sum(p <= eps | p >= 1 - eps)
  lp <- log(pc / (1 - pc))
  lp_var <- stats::var(lp)
  slope_estimable <- is.finite(lp_var) && lp_var > 0

  cal_int <- NA_real_; cal_int_lo <- NA_real_; cal_int_hi <- NA_real_
  cal_slope <- NA_real_; cal_slope_lo <- NA_real_; cal_slope_hi <- NA_real_
  citl <- NA_real_; citl_lo <- NA_real_; citl_hi <- NA_real_
  slope_note <- ""
  p_platt <- rep(base_rate, final_rows)

  citl_fit <- suppressWarnings(try(
    stats::glm(y ~ 1, family = stats::binomial(), offset = lp), silent = TRUE))
  if (!inherits(citl_fit, "try-error")) {
    cf <- summary(citl_fit)$coefficients
    citl    <- cf[1, 1]
    citl_lo <- cf[1, 1] - 1.959964 * cf[1, 2]
    citl_hi <- cf[1, 1] + 1.959964 * cf[1, 2]
  }

  if (slope_estimable) {
    fit <- suppressWarnings(try(
      stats::glm(y ~ lp, family = stats::binomial()), silent = TRUE))
    if (!inherits(fit, "try-error") && nrow(summary(fit)$coefficients) >= 2) {
      cf <- summary(fit)$coefficients
      cal_int      <- cf[1, 1]
      cal_int_lo   <- cf[1, 1] - 1.959964 * cf[1, 2]
      cal_int_hi   <- cf[1, 1] + 1.959964 * cf[1, 2]
      cal_slope    <- cf[2, 1]
      cal_slope_lo <- cf[2, 1] - 1.959964 * cf[2, 2]
      cal_slope_hi <- cf[2, 1] + 1.959964 * cf[2, 2]
      p_platt <- as.numeric(stats::predict(fit, type = "response"))
    } else {
      slope_note <- paste0("the logistic recalibration did not converge, so the ",
                           "calibration slope is not reported")
    }
  } else {
    slope_note <- paste0("every row carries the same predicted probability, so the ",
                         "calibration slope is not estimable — there is no spread to ",
                         "stretch or shrink")
  }

  slope_reading <- if (is.na(cal_slope)) {
    slope_note
  } else if (cal_slope_hi < 1) {
    paste0("the slope of ", fnum(cal_slope, 3), " sits entirely below 1, so the ",
           "predictions are too extreme(over-confident): high predictions run above ",
           "what actually happens and low predictions run below it")
  } else if (cal_slope_lo > 1) {
    paste0("the slope of ", fnum(cal_slope, 3), " sits entirely above 1, so the ",
           "predictions are too timid(under-confident): they are squeezed toward the ",
           "middle relative to what actually happens")
  } else {
    paste0("the slope of ", fnum(cal_slope, 3), " has a 95% interval that includes 1, ",
           "so the spread of the predictions is consistent with being about right")
  }

Step 9: Isotonic regression by pool-adjacent-violators, implemented here

ux <- sort(unique(p))
  gi <- match(p, ux)
  g_n <- integer(length(ux)); g_s <- numeric(length(ux))
  for (i in seq_along(gi)) {
    g <- gi[i]
    g_n[g] <- g_n[g] + 1L
    g_s[g] <- g_s[g] + y[i]
  }
  iso_group <- pava_fit(g_s / g_n, as.numeric(g_n))
  p_iso <- iso_group[gi]
  iso_steps <- length(unique(round(iso_group, 12)))

  platt_brier <- mean((p_platt - y)^2)
  iso_brier   <- mean((p_iso - y)^2)
  platt_bins  <- make_bins(p_platt, y, seq(0, 1, length.out = k_main + 1L))
  iso_bins    <- make_bins(p_iso,   y, seq(0, 1, length.out = k_main + 1L))
  platt_ece   <- bin_errors(platt_bins, final_rows, reliable_min)$ece
  iso_ece     <- bin_errors(iso_bins,   final_rows, reliable_min)$ece

Step 10: Discrimination — rank AUC (ties get average ranks)

rk  <- rank(p)
  auc <- (sum(rk[y == 1L]) - as.numeric(n1) * (n1 + 1) / 2) /
         (as.numeric(n1) * as.numeric(n0))
  auc_band_txt <- auc_band(auc)

Step 11: Binning sensitivity — does the verdict survive re-binning?

k_half <- max(3L, as.integer(k_main %/% 2L))
  k_dbl  <- min(20L, as.integer(k_main * 2L))
  ew_ks  <- sort(unique(c(k_half, k_main, k_dbl)))
  binning_rows <- list()
  for (kk in ew_ks) {
    bb <- make_bins(p, y, seq(0, 1, length.out = kk + 1L))
    ee <- bin_errors(bb, final_rows, reliable_min)
    binning_rows[[length(binning_rows) + 1L]] <- data.frame(
      binning  = paste0(kk, " equal-width bins"),
      n_bins   = ee$n_bins,
      ece      = round(ee$ece, 4),
      mce      = round(ee$mce_reliable, 4),
      verdict  = ece_band(ee$ece),
      stringsAsFactors = FALSE
    )
  }
  q_breaks <- unique(as.numeric(stats::quantile(
    p, probs = seq(0, 1, length.out = k_main + 1L), type = 7, na.rm = TRUE)))
  q_breaks[1] <- min(q_breaks[1], 0); q_breaks[length(q_breaks)] <- max(q_breaks[length(q_breaks)], 1)
  qb <- make_bins(p, y, q_breaks)
  qe <- bin_errors(qb, final_rows, reliable_min)
  binning_rows[[length(binning_rows) + 1L]] <- data.frame(
    binning  = paste0(max(1L, length(q_breaks) - 1L), " equal-count bins"),
    n_bins   = qe$n_bins,
    ece      = round(qe$ece, 4),
    mce      = round(qe$mce_reliable, 4),
    verdict  = ece_band(qe$ece),
    stringsAsFactors = FALSE
  )
  binning_df <- do.call(rbind, binning_rows)
  rownames(binning_df) <- NULL
  verdicts <- unique(binning_df$verdict)
  binning_agrees <- length(verdicts) == 1L
  ece_spread <- max(binning_df$ece, na.rm = TRUE) - min(binning_df$ece, na.rm = TRUE)

Distribution of the predicted probabilities — where the model actually lives.

hist_k <- min(20L, max(5L, as.integer(floor(final_rows / 5))))
  hb <- make_bins(p, y, seq(0, 1, length.out = hist_k + 1L))
  hist_df <- data.frame(
    bucket = paste0(fnum(hb$bin_lo, 2), " to ", fnum(hb$bin_hi, 2)),
    count  = hb$n,
    share  = round(hb$n / final_rows, 4),
    stringsAsFactors = FALSE
  )
  h_ok <- hb$n
  busiest_i <- which.max(h_ok)
  busiest_bucket <- hist_df$bucket[busiest_i]
  busiest_share  <- hist_df$share[busiest_i]
  mid_share <- sum(hb$n[hb$bin_lo >= 0.2 & hb$bin_hi <= 0.8]) / final_rows
  edge_share <- sum(hb$n[hb$bin_hi <= 0.1 | hb$bin_lo >= 0.9]) / final_rows

  remedy_df <- data.frame(
    method = c("As delivered(no correction)",
               "Platt scaling(logistic recalibration)",
               "Isotonic regression(pool-adjacent-violators)"),
    brier = round(c(brier, platt_brier, iso_brier), 5),
    ece   = round(c(ece, platt_ece, iso_ece), 4),
    note = c(
      "The predicted probabilities exactly as supplied.",
      paste0("The fitted intercept and slope applied back to the predicted logit. It can ",
             "shift and stretch the probabilities but cannot change their ORDER, so it ",
             "leaves discrimination untouched. Fitted on these same rows."),
      paste0("A non-decreasing step function fitted by pooling adjacent violators, here ",
             format(iso_steps, big.mark = ","), " distinct steps. It is far more flexible ",
             "than Platt scaling and is therefore the most optimistic in-sample.")),
    stringsAsFactors = FALSE
  )

  discrim_df <- data.frame(
    metric = c("AUC(discrimination)", "Observed base rate", "Brier score",
               "Brier score of a base-rate-only forecast", "Brier skill score"),
    value = c(fnum(auc, 3), fpct(base_rate), fnum(brier, 5), fnum(unc, 5),
              fnum(bss, 4)),
    interpretation = c(
      paste0("Probability that a random &#x27;", positive_label, "' row is scored above a random '",
             negative_label, "&#x27; row — ", auc_band_txt,
             " ranking. 0.500 means no ranking ability at all."),
      paste0("The share of rows that were &#x27;", positive_label, "'."),
      "The delivered probabilities scored against the outcomes.",
      paste0("What you would score by predicting ", fpct(base_rate),
             " for every single row. That forecast is perfectly calibrated and ranks nothing."),
      paste0("How much the delivered probabilities beat that base-rate-only forecast; ",
             "0 means no better, 1 would be perfect.")),
    stringsAsFactors = FALSE
  )

Step 14: KPI metrics + machine channels

metrics <- list(
    `Observations`        = final_rows,
    `Calibration`         = cal_verdict,
    `ECE`                 = round(ece, 4),
    `Calibration Slope`   = if (is.na(cal_slope)) "not estimable" else round(cal_slope, 3),
    `Brier Score`         = round(brier, 4),
    `AUC(Discrimination)`= round(auc, 3)
  )

  calibration_summary <- list(
    n = final_rows, base_rate = base_rate, mean_pred = mean_pred,
    cal_large = cal_large, brier = brier, reliability = rel, resolution = res,
    uncertainty = unc, residual = resid, bss = bss,
    ece = ece, mce_all = mce_all, mce_reliable = mce_reliable,
    n_bins = n_bins_used, k_main = k_main, reliable_min = reliable_min,
    thin_bins = thin_bins,
    cal_int = cal_int, cal_int_lo = cal_int_lo, cal_int_hi = cal_int_hi,
    cal_slope = cal_slope, cal_slope_lo = cal_slope_lo, cal_slope_hi = cal_slope_hi,
    citl = citl, slope_estimable = slope_estimable,
    auc = auc, auc_band = auc_band_txt,
    platt_brier = platt_brier, iso_brier = iso_brier,
    platt_ece = platt_ece, iso_ece = iso_ece, iso_steps = iso_steps,
    n_clamped = n_clamped, binning_agrees = binning_agrees,
    ece_spread = ece_spread, verdict = cal_verdict,
    positive_label = positive_label, negative_label = negative_label
  )

  json_output <- list(
    answer = paste0(
      "Calibration of ", score_name, " against ", actual_name, " (positive = &#x27;",
      positive_label, "&#x27;) across ", format(final_rows, big.mark = ","),
      " rows: the predictions are ", cal_verdict, ", with an expected calibration error of ",
      fnum(ece, 4), " over ", n_bins_used, " bins. ", direction, ". ",
      "The Brier score of ", fnum(brier, 4), " splits into reliability ", fnum(rel, 4),
      ", resolution ", fnum(res, 4), " and uncertainty ", fnum(unc, 4),
      " — the reliability part is what a recalibration could remove. ",
      if (is.na(cal_slope))
        paste0("The calibration slope is not estimable because ", slope_note, ". ")
      else
        paste0("The logistic recalibration gives an intercept of ", fnum(cal_int, 3),
               " and a slope of ", fnum(cal_slope, 3), " (1.000 is ideal): ",
               slope_reading, ". "),
      binning_sentence, " ", discrim_sentence
    ),
    cards = lapply(
      c("tldr", "overview", "preprocessing", "reliability_curve", "calibration_bins",
        "calibration_fit", "brier_decomposition", "score_histogram",
        "binning_sensitivity", "recalibration", "discrimination"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
    actual_name = actual_name, score_name = score_name,
    positive_label = positive_label, negative_label = negative_label,
    positive_rule = positive_rule, scale_rule = scale_rule,
    n1 = n1, n0 = n0, base_rate = base_rate, mean_pred = mean_pred, cal_large = cal_large,
    brier = brier, rel = rel, res = res, unc = unc, resid = resid, bss = bss,
    ece = ece, mce_all = mce_all, mce_reliable = mce_reliable,
    n_bins_used = n_bins_used, k_main = k_main, reliable_min = reliable_min,
    thin_bins = thin_bins,
    cal_int = cal_int, cal_slope = cal_slope, citl = citl,
    cal_slope_lo = cal_slope_lo, cal_slope_hi = cal_slope_hi,
    slope_estimable = slope_estimable, slope_note = slope_note,
    slope_reading = slope_reading,
    auc = auc, auc_band_txt = auc_band_txt,
    platt_brier = platt_brier, iso_brier = iso_brier,
    platt_ece = platt_ece, iso_ece = iso_ece, iso_steps = iso_steps,
    n_clamped = n_clamped, binning_agrees = binning_agrees, ece_spread = ece_spread,
    cal_verdict = cal_verdict, direction = direction,
    binning_sentence = binning_sentence, thin_sentence = thin_sentence,
    worst_sentence = worst_sentence, discrim_sentence = discrim_sentence,
    insample_sentence = insample_sentence, clamp_sentence = clamp_sentence,
    busiest_bucket = busiest_bucket, busiest_share = busiest_share,
    mid_share = mid_share, edge_share = edge_share,
    bins_main = bins_main,
    reliability_df = reliability_df, bin_table_df = bin_table_df,
    brier_df = brier_df, fit_df = fit_df, hist_df = hist_df,
    binning_df = binning_df, remedy_df = remedy_df, discrim_df = discrim_df,
    calibration_summary = calibration_summary, metrics = metrics,
    json_output = json_output
  )
}

This is the exact R code executed to produce the analysis above. Run it with the same dataset to reproduce these results.

Your data has more stories to tell. Run any analysis on your own data — validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No Signup Sign Up Free
Something wrong? Report it and we'll fix it — or get your credits back
Download PDF · Print · Share ·Refresh this analysis ·Edit this report

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing