---
title: "Which ICC do you actually need?"
subtitle: "Intraclass correlation, decided. Companion analysis for the MCP Analytics video"
author: "MCP Analytics"
date: "2026-08-15"
output:
  html_document:
    theme: flatly
    toc: true
    toc_float: true
    code_folding: show
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
library(psych)
```

## What ICC is for

The intraclass correlation coefficient answers one family of questions: **can I trust
a measurement that passes through people or instruments?** Whenever the same thing is
measured more than once (by two raters, two devices, or the same instrument on two
occasions), the measurements will not agree perfectly. ICC quantifies how much of the
variation in your data reflects *real differences between the things being measured*,
versus noise introduced by *the measuring itself*.

It is **not a prediction**. It is a verdict on the measuring, and every analysis you
run downstream inherits that verdict. A driver analysis, a group comparison, or a
forecast built on unreliable scores is quietly built on sand; ICC is how you check
the foundation first.

### Common applications

| Field | The question ICC answers |
|---|---|
| Clinical research | Do two radiologists read the same scan the same way? (inter-rater reliability) |
| Psychometrics & surveys | Does the instrument give the same score on a retest? (test–retest reliability) |
| Machine learning | Can I rely on the labels my annotation team produced? (label-quality audit) |
| Quality control | Do two gauges, labs, or operators agree on the same parts? (method agreement) |
| People operations | Do interviewers or QA reviewers score candidates/calls consistently? |

## What you need as input

One table, **wide format**: a row per subject (the things being measured), a column
per rater or measurement occasion, numeric scores in the cells.

- **Scores** should be continuous or at least ordered with enough distinct values to
  treat as numeric. (Binary yes/no ratings are better served by kappa and McNemar; ICC variants for
  dichotomous data exist, but kappa is the standard there.)
- **Design matters more than size**: know whether the *same* raters scored every
  subject (a crossed design) or each subject got different raters. That is Question
  1 of the chooser below.
- **Size still matters**: confidence intervals with a dozen subjects are wide, as
  this example will demonstrate honestly. Koo & Li (2016) suggest at least 30
  subjects and 3 raters for stable estimates; below that, report the interval and
  let it do the talking.

## What you get as output

One number per ICC form, usually between 0 and 1 (sample estimates can dip below
zero when reliability is truly poor), plus a 95% confidence interval and an F-test.
The conventional interpretation bands (Koo & Li 2016), applied to the **interval**,
not just the point estimate:

| ICC | Reliability |
|---|---|
| < 0.50 | Poor |
| 0.50 – 0.75 | Moderate |
| 0.75 – 0.90 | Good |
| > 0.90 | Excellent |

## The six forms, and the three questions that choose between them

The Shrout & Fleiss (1979) lineup has six forms: three models × single-or-average
measures. Three questions select yours:

1. **Did the same raters score every subject?**
   *No* → one-way model, **ICC(1)**. Rater habits cannot be separated from noise.
   *Yes* → two-way model; continue.
2. **Do rater differences in level count as error?** If a 70 from one rater must mean
   what a 70 means from another (scores travel between raters), you need **absolute
   agreement**: ICC(2,1). If only ranking matters and level washes out, you need
   **consistency**: ICC(3,1). (Statisticians file the 2-vs-3 choice under whether raters are a random
   sample or a fixed panel; McGraw & Wong 1996 decouple the axes. For picking the
   number, agreement versus consistency is what decides it.)
3. **Will decisions use a single rater's score, or the panel average?**
   Single → ICC(·,1). Average of k raters → ICC(·,k), which is always at least as
   high whenever there is real reliability to amplify.

## The worked example

### The data and its provenance

Twelve subjects, each scored by the same three raters. The dataset is synthetic and
fully reproducible: generated by `icc_example.py` (seed 42) with true subject scores
~ N(70, 10), additive rater biases of 0, +10, and −3 points, and rating noise
~ N(0, 3). Because we built it, we know the ground truth the estimators are chasing,
which is the point of a worked example.

```{r data}
ratings <- read.csv("ratings.csv")
ratings
```

### First look: the raters do not agree on level

```{r eda}
colMeans(ratings)
matplot(ratings, type = "b", pch = 16, lty = 1,
        xlab = "Subject", ylab = "Score",
        main = "Three raters, twelve subjects")
legend("topright", legend = names(ratings), col = 1:3, pch = 16, bty = "n")
```

Rater 2 runs about `r round(mean(ratings$rater2) - mean(ratings$rater1), 1)` points
hotter than rater 1 on the same subjects: visibly parallel profiles, shifted in
level. Whether that shift counts as *error* is exactly what separates the ICC forms.

### Every form at once

```{r icc}
res <- ICC(ratings)
res$results
```

### Reading the table against the design

Our design: the **same panel** rated everyone (two-way), scores need to **travel
between raters** (absolute agreement), and decisions ride on **single ratings**
(single measure). The three questions select **ICC(2,1)**:

```{r icc21}
icc21 <- res$results[res$results$type == "ICC2", ]
icc21
```

Point estimate `r round(icc21$ICC, 2)`, 95% CI
[`r round(icc21$"lower bound", 2)`, `r round(icc21$"upper bound", 2)`]. With n = 12
subjects the interval is wide, and per Koo & Li the interval, not the point, should
drive the label.

The consistency form on the identical data:

```{r icc31}
icc31 <- res$results[res$results$type == "ICC3", ]
round(c(ICC = icc31$ICC, lower = icc31$"lower bound", upper = icc31$"upper bound"), 3)
```

Same 36 numbers: agreement `r round(icc21$ICC, 2)`, consistency
`r round(icc31$ICC, 2)`. Neither is wrong; they answer different questions.

### The calibration experiment: the gap is a diagnosis, not a verdict

If the agreement/consistency gap is driven by rater *level* bias, then removing each
rater's mean offset should send agreement up toward the consistency value. Center
each rater and recompute:

```{r calibrate}
centered <- sweep(ratings, 2, colMeans(ratings)) + mean(as.matrix(ratings))
res_c <- ICC(centered)
res_c$results[res_c$results$type %in% c("ICC2", "ICC3"),
              c("type", "ICC", "lower bound", "upper bound")]
```

After calibration, agreement ICC(2,1) rises to
`r round(res_c$results[res_c$results$type=="ICC2","ICC"], 2)`, effectively the
consistency ceiling. The gap between the two forms *measured the level bias* and told
us the fix (calibrate raters), not the verdict (distrust the instrument).

### Averaging: the panel is more reliable than any member

```{r averages}
res$results[res$results$type %in% c("ICC2k", "ICC3k"),
            c("type", "ICC", "lower bound", "upper bound")]
```

If decisions will only ever use the three-rater average, ICC(2,k) applies. But
report the form you will actually live with downstream.

### Cross-check against an independent implementation

The video's on-screen numbers were computed independently in Python (ANOVA mean
squares, no libraries). Agreement between implementations:

```{r crosscheck}
manual <- c(ICC1 = 0.57, ICC2 = 0.62, ICC3 = 0.95, ICC2k = 0.83, ICC3k = 0.98)
r_vals <- round(res$results[match(c("ICC1","ICC2","ICC3","ICC2k","ICC3k"),
                                  res$results$type), "ICC"], 2)
data.frame(form = names(manual), python = as.numeric(manual), R = r_vals,
           match = as.numeric(manual) == r_vals)
```

## Honest limitations

- **Small samples**: our n = 12 produces the CI [0.04, 0.89] you saw above, nearly
  the whole ruler. Report intervals, and prefer ≥30 subjects when you design the study.
- **Dependence and structure**: strongly dependent ratings, nested designs, or
  missing cells push you toward mixed-effects formulations rather than the closed
  forms shown here.
- **Negative estimates** happen in real samples when reliability is truly poor; they
  are a finding, not a bug.
- **ICC is scale-dependent**: restrict the range of subjects and ICC drops even with
  identical raters, so compare ICCs only across comparable populations.

## References

- Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: uses in assessing
  rater reliability. *Psychological Bulletin*, 86(2), 420–428.
- McGraw, K.O. & Wong, S.P. (1996). Forming inferences about some intraclass
  correlation coefficients. *Psychological Methods*, 1(1), 30–46.
- Koo, T.K. & Li, M.Y. (2016). A guideline of selecting and reporting intraclass
  correlation coefficients for reliability research. *Journal of Chiropractic
  Medicine*, 15(2), 155–163.

## Session

```{r session}
sessionInfo()$R.version$version.string
packageVersion("psych")
```