# Intraclass correlation (ICC): agent brief

> **Audience: an agent deciding whether to run this analysis on a dataset.** Not marketing
> copy. Terse, complete, and honest about failure. If you are asked "should I run an ICC
> here", you should be able to answer from this file alone, **including saying no**.
>
> Tool: `standard_icc` · free page: `/free/standard_icc` ·
> worked example: `/worked-examples/icc` · lesson: https://youtu.be/Ff7cBBh9oSU

---

## 1. What it answers

**How much of a score is the thing being scored, and how much is the person scoring it.**

Two people rate the same work and disagree. ICC splits the variation into the part that
comes from real differences between subjects and the part that comes from the raters. A high
ICC means the score travels: swap the rater and you get roughly the same number.

**The distinguishing feature is that the same subjects are rated by more than one rater**,
and you care whether the measurement instrument (often a person) is trustworthy. If you care
whether two *methods* agree at the level of an individual pair, that is a different question.

**Questions it is mistaken for:**

| Actually asked | Right tool |
|---|---|
| *How far apart can any single pair of measurements be?* | `standard_bland_altman`. ICC can be 0.99 while individual pairs differ by a clinically fatal amount |
| *Do the two raters agree on a yes/no call?* | `standard_mcnemar` (paired binary), or Cohen's kappa |
| *Is rater A higher on average than rater B?* | that is a mean difference, not reliability. `standard_group_comparison` |
| *Do these two variables move together?* | Pearson correlation. Correlation is indifferent to a constant offset; ICC is not |
| *Is this one score unusual?* | `standard_anomaly_detection` |

## 2. When it applies, and when it does not

**Apply it when: multiple raters scored the same subjects, on a numeric scale, and you want
to know whether the scores are a property of the subject rather than the rater.**

**Do not apply it when:**

- **Each subject was rated once.** There is no rater variance to separate, and nothing to
  compute.
- **The outcome is categorical.** ICC needs a numeric scale. Use kappa for agreement on
  labels.
- **Raters scored *different* subjects.** The design has to be crossed or at least
  overlapping; otherwise rater and subject effects are inseparable.
- **You actually need limits of agreement.** ICC answers "does the score rank subjects
  consistently", not "how wrong can one measurement be". Those come apart badly, and §5 has
  the numbers.

## 3. What the data must look like

`column_mapping` requires `subject`, `rater`, `score`. **Long format, one row per rating.**

| key | type | meaning |
|---|---|---|
| `subject` | any id | what was rated |
| `rater` | categorical | who or what produced the rating |
| `score` | numeric | the rating value |

**Wide format does not work** and this is the most common blocker: a spreadsheet with one
row per subject and one column per rater has to be reshaped to long first. Note also that
**auto-mapping did not engage even on exact column-name matches** in the recorded run, so
pass `column_mapping` explicitly rather than relying on inference.

## 4. What it returns, and how to read each piece

| Output | Read it as | The trap |
|---|---|---|
| **ICC results table** | all six Shrout-Fleiss forms, each with a 95% interval | **six numbers, one is yours.** They disagree by a lot on the same data. Picking the flattering one is the single most common abuse of this analysis |
| **Which one to report** | the form matching your design and intent | it depends on three choices: one-way or two-way, absolute agreement or consistency, single rater or the mean of k. Answer those before looking at any number |
| **Systematic rater bias** | whether a rater is consistently high or low | consistency forms **forgive** a constant offset; absolute-agreement forms do not. A rater who is always 3 points high scores well on one and badly on the other |
| **Subject-level agreement** | where raters diverged most | a diagnostic, not a verdict |
| **Where the disagreement comes from** | the variance split itself | this is the actual answer to the business question |
| **95% confidence interval** | the plausible range for the ICC | **judge the Koo-Li band by the interval, not the point.** A point estimate of 0.95 with an interval of 0.12 to 0.96 is not "excellent" |
| **AI insight** | plain-language verdict | reads the numbers it was given. It cannot know your design was one-way |

## 5. How it fails

**The headline failure is picking the wrong form and reporting it as "the ICC".** There is
no such thing as the ICC. On the worked example the six forms run from **0.573 to 0.983** on
one dataset, and every one is correctly computed.

**Failure modes that yield a plausible wrong answer rather than an error:**

- **Reporting an average-of-k form when a single rater will do the work.** `ICC(3,k)` is
  0.983 on data where `ICC(1,1)` is 0.573. If one person rates each case in production, the
  0.983 describes a workflow you do not run.
- **Using a consistency form when absolute agreement is what matters.** Consistency ignores
  systematic offsets, so a rater who is reliably wrong looks reliable.
- **Reading the point estimate against the Koo-Li thresholds** (poor/moderate/good/excellent)
  while ignoring an interval that spans three of those bands.
- **High ICC read as interchangeable measurements.** It is not. See below.
- **Wide-format data reshaped wrongly**, silently pairing the wrong rater with the wrong
  subject.

**The one to say out loud: a high ICC does not mean two methods can be swapped.** ICC and
Bland-Altman are the textbook complement precisely because a correlation of 0.986 sat beside
limits of agreement of roughly ±4.5 kg on the same paired data. Excellent ranking, and a
gap no clinician would accept on an individual patient.

## 6. Verified numbers you may cite

From the LAT-2246 worked example (12 subjects x 3 raters, seed 42), agreed four ways: the
notebook, `psych::ICC`, an independent numpy recompute, and an adversarial re-derivation.

| form | value | 95% CI |
|---|---|---|
| ICC(1,1) | 0.573 | 0.24 – 0.83 |
| ICC(2,1) | 0.623 | 0.04 – 0.89 |
| ICC(3,1) | 0.951 | 0.88 – 0.98 |
| ICC(2,k) | 0.832 | 0.12 – 0.96 |
| ICC(3,k) | 0.983 | 0.96 – 0.99 |

**Do not say** any single one of these is "the reliability" of the data. The spread from
0.573 to 0.983 *is* the lesson.

**Look at ICC(2,k):** the point estimate is 0.832 and the interval runs 0.12 to 0.96. That
one number would be reported as "good" and its interval covers "poor" through "excellent".

**Live-tool reproduction: RUN AND MATCHED.** `standard_icc` through the free-analyze door,
run `free_kbWaQdevqXTj3G4k_75cffed30db0` -> `rpt_v6r7mm6QUzBga89H_daRGVBqaJCDai3D`, returned
all five point estimates matching numpy, `psych::ICC` and the adversarial recompute **to
three decimal places**. Unlike most of our worked examples this one was opened and compared,
so the reproduction is checked rather than merely attested.

**A separate benchmark run** on the carData Davis set (182 subjects, 2 methods, 364 ratings)
returned ICC(2,1) = 0.986 against a numpy two-way random-effects recompute of 0.9855. Exact.

## 7. Where everything is

| | |
|---|---|
| Tool | `standard_icc` |
| Free page | https://mcpanalytics.ai/free/standard_icc |
| Worked example | https://mcpanalytics.ai/worked-examples/icc |
| Notebook | `/worked-examples/files/icc.Rmd` |
| Validation record | `lattice/v2/refs/LAT-2246-edu-videos-stats-choosers/worked-example/VALIDATION.md` |
| Our tool's own run | `rpt_v6r7mm6QUzBga89H_daRGVBqaJCDai3D` (free-analyze, 2026-08-15, opened and compared) |
| Deliberate complement | `standard_bland_altman` on the same paired data |

## 8. A note on the intervals

The deployed tool did **not** report confidence intervals until 2026-08-15 (LAT-2249). They
were added to the deployed R with Satterthwaite for ICC(2,1) and a Spearman-Brown bound
transform for the (,k) forms, validated against `psych::ICC` to 1.4e-06 before deploy, then
the same dataset was re-run through the live tool and every form carried its interval.

That gap is worth knowing because it is the class of defect only a completeness contract
catches: the tool computed five correct numbers and omitted the five ranges that decide
whether any of them means anything. Every other gate passed.

## 9. Routing shortcut

```
Were the same subjects rated by MORE THAN ONE rater, on a numeric scale?
├── no, one rating each      → NOT this. No rater variance to separate.
├── ratings are categorical  → NOT this. Use kappa.
└── yes
    ├── do you need "how far apart can one pair be?"  → standard_bland_altman, not this
    ├── data in wide format? → reshape to long first: subject, rater, score
    └── answer THREE questions before reading any number:
        one-way or two-way? · absolute agreement or consistency? · single rater or mean of k?
        then report THAT form, with its interval, and say which one you picked.
```