Which ICC do you actually need?

The data

Twelve subjects rated by three raters. The same twelve rows, scored by every ICC form, cross-checked against numpy, R's psych::ICC, and an independent recompute, agreeing to three decimal places.

The headline numbers

• ICC(1,1)0.573
• ICC(2,1)0.623
• ICC(3,1)0.951
• ICC(2,k)0.832
• ICC(3,k)0.983

One dataset returns 0.573 or 0.983 depending only on which ICC form you ask for. That is not a rounding difference, it is the gap between poor and excellent. The form is a modelling choice about what you are generalising to, not a formula preference.

Download and re-run it

Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.

• icc.html · The knitted document1.4 MB
• icc.Rmd · R Markdown source9 KB
• ratings.csv · The ratings1 KB
• icc_example.py · Generates the data2 KB

The honest limit

A point estimate without its confidence interval is not a reliability claim, and with a dozen subjects those intervals are wide. The document says so rather than quietly reporting three decimal places. The product's own report was missing intervals when this was written, which we filed against ourselves and fixed.

The brief for an AI agent

If an AI assistant is choosing a method for you, it needs different things than you do: the discriminating condition, the questions this gets mistaken for, the failure modes that return a plausible number instead of an error, and figures it can cite rather than generate. We publish that as its own document, written to one bar: an agent asked whether to run this analysis should be able to answer from it alone, including saying no.

Read the agent brief

Where this came from