Intraclass correlation

The method, in full

The practical guide behind this tool: when it applies, how to read the output, and the traps that make it say the wrong thing.

Read the guide →

A simulated example, with the code and its output

The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.

Question: How consistent are ratings of the same subjects across raters, and do some raters run high or low?

Inputs: Simulated data, not real ratings. Seed 1 makes 60 subjects each rated by the same 4 raters. Built in by construction: true subject scores with standard deviation 2.0 around 10, fixed rater offsets of +1.0, +0.3, -0.4 and -0.9, and random noise with standard deviation 1.5. That targets ICC(2,1) = 4.0 / (4.0 + 0.687 + 2.25) = 0.577 (rater-offset variance 2.06 / 3 = 0.687) and ICC(3,1) = 4.0 / (4.0 + 2.25) = 0.640.

Method: Intraclass correlation from a two-way analysis of variance table (subjects by raters), using the Shrout and Fleiss forms. ICC(1,1) ignores which rater gave which score. ICC(2,1) treats raters as a random sample and counts rater bias as error. ICC(3,1) treats these raters as the only ones of interest and ignores rater bias. ICC(2,k) and ICC(3,k) are the matching figures for the average of the k raters. A bootstrap interval re-draws the 60 subjects with replacement 1,000 times; the middle 95% of the recomputed values is the interval. Rater means show who runs high or low.

Result: ICC(1,1) is 0.475, ICC(2,1) is 0.492 and ICC(3,1) is 0.562. The average-of-4 forms are higher: ICC(2,k) 0.795 and ICC(3,k) 0.837. The bootstrap 95% interval is 0.299 to 0.624 for ICC(2,1) and 0.369 to 0.692 for ICC(3,1), so one run of 60 subjects is wide around the built-in 0.577 for ICC(2,1) and 0.640 for ICC(3,1). Rater means are 10.80, 10.08, 9.01 and 9.02, which recover the order of the built-in offsets. ICC(3,1) is above ICC(2,1) because it ignores that rater bias.

Limit: The example uses complete data, equal numbers of ratings per subject, continuous scores and noise that is the same for every rater. It does not show missing ratings, ordinal scales, raters who differ in spread, or which forms and intervals the tool itself reports.

Python script (numpy and scipy only):

import numpy as np
# Simulated data. Built in: 60 subjects, 4 raters. Subject true-score SD 2.0, fixed rater offsets +1.0, +0.3, -0.4, -0.9
# (rater offsets +1.0, +0.3, -0.4, -0.9 fixed), noise SD 1.5.
rng = np.random.default_rng(1)
n, k = 60, 4
subj = rng.normal(0, 2.0, n)
bias = np.array([1.0, 0.3, -0.4, -0.9])
Y = 10 + subj[:, None] + bias[None, :] + rng.normal(0, 1.5, (n, k))

def iccs(Y):
    n, k = Y.shape
    g = Y.mean()
    ssr = k * ((Y.mean(1) - g) ** 2).sum()
    ssc = n * ((Y.mean(0) - g) ** 2).sum()
    sst = ((Y - g) ** 2).sum()
    sse = sst - ssr - ssc
    msr = ssr / (n - 1); msc = ssc / (k - 1); mse = sse / ((n - 1) * (k - 1))
    msw = (ssc + sse) / (n * (k - 1))
    return {"ICC(1,1)": (msr - msw) / (msr + (k - 1) * msw),
            "ICC(2,1)": (msr - mse) / (msr + (k - 1) * mse + k * (msc - mse) / n),
            "ICC(3,1)": (msr - mse) / (msr + (k - 1) * mse),
            "ICC(2,k)": (msr - mse) / (msr + (msc - mse) / n),
            "ICC(3,k)": (msr - mse) / msr}

est = iccs(Y)
for key, v in est.items(): print("%s %.3f" % (key, v))
B = [iccs(Y[rng.integers(0, n, n)]) for _ in range(1000)]
for key in ("ICC(2,1)", "ICC(3,1)"):
    lo, hi = np.percentile([b[key] for b in B], [2.5, 97.5])
    print("%s bootstrap 95%% interval %.3f to %.3f" % (key, lo, hi))
print("rater means:", np.round(Y.mean(0), 2), " built-in offsets from 10:", bias)
print("target ICC(2,1) by construction: %.3f" % (4.0 / (4.0 + (bias ** 2).sum() / (k - 1) + 2.25)))
print("target ICC(3,1) by construction: %.3f" % (4.0 / (4.0 + 2.25)))

Printed output from running the code above:

ICC(1,1) 0.475
ICC(2,1) 0.492
ICC(3,1) 0.562
ICC(2,k) 0.795
ICC(3,k) 0.837
ICC(2,1) bootstrap 95% interval 0.299 to 0.624
ICC(3,1) bootstrap 95% interval 0.369 to 0.692
rater means: [10.8  10.08  9.01  9.02]  built-in offsets from 10: [ 1.   0.3 -0.4 -0.9]
target ICC(2,1) by construction: 0.577
target ICC(3,1) by construction: 0.640

What's in the report?

How reliable are these ratings across raters or repeat measurements? Intraclass correlations with their ranges, and which raters run high or low.