Standard Icc
Executive Summary

Executive Summary

Rater reliability across 30 Sample ID values and 4 raters

Subjects
30
Raters
4
Ratings Used
120
ICC(2,1)
0.782
Reliability
good
F-test p
8.36e-29
The headline reliability is ICC(2,1) = 0.782 — GOOD absolute agreement (Koo & Li) for a single rater's Quality Score across 30 Sample ID values and 4 raters. That the raters distinguish Sample ID values at all is statistically confirmed (F = 21.91, p = 8.36e-29). Which ICC to report: use ICC(2,1) if your raters are a random sample and absolute Quality Score values matter (the safest default); ICC(3,1) = 0.839 if these exact 4 raters are the only ones of interest; ICC(2,k) = 0.935 if decisions rest on the average of all 4 raters rather than any single one.
Interpretation

Your raters show GOOD absolute agreement (ICC(2,1) = 0.782) on a single rater basis—well above the 0.75 threshold. The F-test (F = 21.91, p = 8.36e-29) confirms statistically that raters distinguish Sample ID values meaningfully. Choose ICC(2,1) if raters are a random sample and absolute scores matter (safest default); ICC(3,1) = 0.839 if only these 4 raters will ever be used; ICC(2,k) = 0.935 if decisions rest on the panel average. Rater reliability is solid, but systematic bias and residual noise leave room for improvement through calibration and clearer rubrics.

Overview

Analysis Overview

Intraclass correlation from 120 ratings: 30 Sample ID values x 4 raters.

N Subjects30
N Raters4
N Ratings120
Icc 2 10.782
Interpretation

Intraclass correlation quantifies how reliably your raters measure real differences in Quality Score across 30 Sample ID values. With 120 ratings from 4 raters, the choice of ICC form depends on two decisions: whether you care about a single rater's score (ICC(x,1)) or the panel average (ICC(x,k)), and whether absolute values must match (ICC(2,1)) or only relative ordering (ICC(3,1)). The gap between ICC(3,1) = 0.839 and ICC(2,1) = 0.782 (0.058 points) reveals that systematic Rater Name differences—leniency and severity bias—meaningfully lower absolute agreement. For most purposes, report ICC(2,1) = 0.782 as your headline: it reflects the reliability of one rater working alone with absolute scores, and is the most conservative estimate.

Data Preparation

Data Quality

Ratings used, duplicates averaged, incomplete subjects excluded.

Initial Rows120
Final Rows120
Rows Removed0
N Duplicates0
N Incomplete Subjects0
Interpretation

All 120 ratings were retained in the final analysis; no rows were removed and no duplicate ratings were found. Every Sample ID value was rated by all 4 raters, forming a complete 30 × 4 crossing. This completeness is essential for two-way ANOVA-based ICC computation and eliminates confounding from missing data patterns. The consistency of Sample ID identification across raters ensured clean data entry, and the absence of duplicates means each rating was counted exactly once. This clean structure supports confidence in the ICC estimates.

Data Table

ICC Results — Which One to Report

All ICC forms with interpretation bands and when to use each.

Icc TypeValueInterpretationUse When
ICC(1,1) — one-way random, single rater0.778goodEach Sample ID is rated by a DIFFERENT random set of raters (no shared panel).
ICC(2,1) — absolute agreement, single rater0.782goodRaters are a random sample and absolute Quality Score values must match — the default to report.
ICC(3,1) — consistency, single rater0.839goodThese exact 4 raters are the only ones of interest; systematic differences between them are forgiven.
ICC(2,k) — absolute agreement, average of all raters0.935excellentDecisions rest on the AVERAGE Quality Score of all 4 raters, and absolute values matter.
ICC(3,k) — consistency, average of all raters0.954excellentDecisions rest on the average of these exact 4 raters; only relative ordering matters.
Interpretation

Five ICC forms span the design space. ICC(2,1) = 0.782 (GOOD) is the default: it assumes raters are a random sample and absolute Quality Score values must match. ICC(3,1) = 0.839 (GOOD) is higher because it forgives systematic rater offsets; use it only if these exact 4 raters are fixed. The 0.058 gap between them quantifies the cost of rater bias. For panel decisions, ICC(2,k) = 0.935 (EXCELLENT) and ICC(3,k) = 0.954 (EXCELLENT) are much higher because averaging 4 raters cancels individual noise. ICC(1,1) = 0.778 (GOOD) applies only if each subject is rated by a different rater set. All values fall in the GOOD-EXCELLENT range, indicating your raters reliably measure real subject differences.

Visualization

Systematic Rater Bias

Mean Quality Score per rater on the same Sample ID values.

Interpretation

Dr. Adams is the most lenient rater (mean 51.79) and Dr. Chen the most severe (mean 46.92), a spread of 4.87 points on the same 30 subjects. This systematic bias is pure leniency/severity, not a difference in what was rated. The rater bias accounts for 6.9% of total variance. Calibrating raters against a shared standard—anchored examples, explicit thresholds, or consensus discussion—would reduce this gap and raise absolute agreement (ICC(2,1)). Such calibration is feasible and would push reliability higher without requiring changes to the rubric or subjects themselves.

Visualization

Subject-Level Agreement

One point per Sample ID: the first two raters' scores, diagonal = perfect agreement.

Interpretation

Dr. Adams and Dr. Baker correlate at r = 0.876, indicating strong relative agreement. However, points cluster in a band parallel to but offset from the perfect-agreement diagonal, illustrating the ICC(3,1) vs ICC(2,1) distinction: the two raters rank subjects consistently but Adams scores systematically higher. Some subjects show genuine disagreement—notably around Adams 59–62 and Baker 49–57—where the two diverge more than the overall offset would predict. These outliers represent true judgment differences, not just bias. The pattern suggests that while rater ordering is reliable, absolute calibration would improve agreement.

Visualization

Where the Disagreement Comes From

Variance in Quality Score split between subjects, raters, and noise.

Interpretation

Total Quality Score variance decomposes as: between-subject signal 78.2%, between-rater systematic bias 6.9%, and residual noise 15%. The dominance of the between-subject bar (78.2%) explains why reliability is high—raters are detecting real differences. The 6.9% rater bias is the cost of leniency/severity spread; calibration sessions would shrink this. The 15% residual noise reflects one-off disagreements on specific subjects; clearer rubrics, anchored examples, or more detailed guidance would shrink this. Both levers are available: calibration is the quicker win for absolute agreement, while rubric refinement would improve consistency on difficult subjects.

Methodology

Methodology

Statistical methodology and diagnostics for Intraclass Correlation — Rater Reliability

Statistical Method

Intraclass Correlation — Rater Reliability

Standard-library analysis: how consistent are your raters, instruments, or repeated measurements? Upload long-format ratings (one row per rating) and get the full intraclass correlation family — ICC(1,1), ICC(2,1), ICC(3,1) and the average-measure versions — computed from ANOVA variance components, with Koo & Li interpretation bands, a systematic rater-bias check, a subject-level agreement plot, and a variance breakdown showing exactly where the disagreement comes from. Built for reliability studies: inter-rater agreement, test-retest, instrument comparison.

Data
N = 120 observations
Assumptions
  • Data is long format: one row per rating, with a subject id, a rater id, and a numeric score
  • Ratings are on a continuous or near-continuous scale (for categorical ratings use a kappa statistic instead)
  • Two-way ICCs assume every retained subject is rated by every rater; subjects missing ratings are excluded and reported
Limitations
  • Subjects not rated by all raters are excluded from the two-way ICCs (they are counted and named)
  • Duplicate ratings of the same subject by the same rater are averaged before analysis
  • ICC assumes rating errors are roughly normal; heavy-tailed or bounded scales can distort it
  • A high ICC on a homogeneous sample can mask poor agreement — the between-subject variance share shows this
Software & Citation
MCP Analytics · mcpanalytics.ai
Code Appendix

Analysis Code

Complete R source code for this analysis

Intraclass Correlation — Rater Reliability

How consistent are your raters, instruments, or repeated measurements? From long-format ratings (one row per rating) this computes the full ICC family — ICC(1,1), ICC(2,1), ICC(3,1) plus the average-measure versions — from two-way ANOVA mean squares, with Koo & Li interpretation bands, a rater-bias check, a subject-level agreement plot, and a variance breakdown showing where the disagreement comes from.

Why This Method?

The ICC is the standard reliability coefficient for continuous ratings: it asks how much of the total variation comes from real differences between the things being rated rather than from rater disagreement. Computing every common form at once answers the perennial question of WHICH ICC to report for a given study design.

What This Analysis Covers

  • Single-measure and average-measure ICCs, consistency and agreement
  • Koo & Li (2016) interpretation bands
  • Systematic rater bias (mean score per rater)
  • Subject-level agreement between the first two raters
  • Variance decomposition: subjects vs raters vs residual noise

Standard Library

Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {subject, rater, score}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Core Analysis Pipeline

compute_shared <- function(df, params, col_map = list()) {
  # === SHARED EXPORTS ===
  #   initial_rows/final_rows/rows_removed  $ row accounting (ratings)
  #   subj_h / rater_h / score_h  $ humanized names of the three mapped columns
  #   n_subjects / k_raters       $ complete-crossing design dimensions
  #   rater_levels                $ character — the actual rater level values
  #   n_duplicates                $ duplicate ratings averaged
  #   n_incomplete                $ subjects excluded (not rated by all raters)
  #   incomplete_subjects         $ character — the excluded subject ids
  #   icc / icc_p                 $ named list of ICC values + F-test p for ICC(3,1)
  #   icc_results_df   $ data.frame(icc_type, value, interpretation, use_when)
  #   rater_means_df   $ data.frame(rater_name, mean_score)
  #   agreement_df     $ data.frame(rater_1_score, rater_2_score) — complete subjects
  #   agreement_r      $ numeric — Pearson r between the first two raters
  #   variance_df      $ data.frame(source, variance_pct)
  #   headline_band    $ Koo & Li band for ICC(2,1)
  #   metrics / json_output
  # === /SHARED EXPORTS ===

  subj_h  <- humanize_semantic("subject", col_map)
  rater_h <- humanize_semantic("rater", col_map)
  score_h <- humanize_semantic("score", col_map)

Step 1: Validate the mapped columns

initial_rows <- nrow(df)
  for (req in c("subject", "rater", "score")) {
    if (is.null(df[[req]])) {
      stop(sprintf("column_mapping must map the &#x27;%s' column (%s / %s / %s).",
                   req, subj_h, rater_h, score_h))
    }
  }

Step 2: Coerce score numeric (95% rule); clean subject/rater ids

v <- df$score
  if (!is.numeric(v)) {
    conv <- suppressWarnings(as.numeric(as.character(v)))
    n_orig <- sum(!is.na(v) & as.character(v) != "")
    if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
      v <- conv
    } else {
      stop(sprintf("The rating column(%s) must be numeric — fewer than 95%% of its values convert cleanly.",
                   score_h))
    }
  }
  df$score   <- v
  df$subject <- trimws(as.character(df$subject))
  df$rater   <- trimws(as.character(df$rater))
  keep <- !is.na(df$score) & !is.na(df$subject) & df$subject != "" &
          !is.na(df$rater) & df$rater != ""
  n_invalid <- sum(!keep)
  df <- df[keep, , drop = FALSE]
  if (nrow(df) < 10) {
    stop(sprintf("Only %d usable ratings after removing rows with missing %s, %s, or %s — need at least 10.",
                 nrow(df), subj_h, rater_h, score_h))
  }

  rater_levels <- sort(unique(df$rater))
  k <- length(rater_levels)
  if (k < 2) {
    stop(sprintf("ICC needs at least 2 distinct values in the %s column; found %d. Each row should be one rating by one rater.",
                 rater_h, k))
  }
  if (k > 30) {
    stop(sprintf("The %s column has %d distinct values — that looks like an identifier, not a set of raters. Map the column naming who/what gave each rating.",
                 rater_h, k))
  }

Step 3: Average duplicate ratings per subject x rater

agg <- aggregate(score ~ subject + rater, data = df, FUN = mean)
  n_duplicates <- nrow(df) - nrow(agg)

Step 4: Keep subjects rated by ALL raters (two-way ICCs need a

complete crossing); report the incomplete ones

cnt <- table(agg$subject)
  complete_subjects   <- names(cnt)[cnt == k]
  incomplete_subjects <- names(cnt)[cnt < k]
  n_incomplete <- length(incomplete_subjects)
  n <- length(complete_subjects)
  if (n < 5) {
    stop(sprintf("ICC needs at least 5 %s values rated by every %s; only %d of %d have a complete set of %d ratings.",
                 subj_h, rater_h, n, length(cnt), k))
  }
  dat <- agg[agg$subject %in% complete_subjects, , drop = FALSE]
  if (is.na(var(dat$score)) || isTRUE(var(dat$score) == 0)) {
    stop(sprintf("The %s values are constant — reliability cannot be estimated.", score_h))
  }
  final_rows <- nrow(dat)
  rows_removed <- initial_rows - final_rows

  dat$subject_f <- factor(dat$subject)
  dat$rater_f   <- factor(dat$rater, levels = rater_levels)

Step 5: ICCs from ANOVA mean squares

Two-way: aov(score ~ subject + rater) -> MSR (subjects), MSC (raters), MSE

a2  <- aov(score ~ subject_f + rater_f, data = dat)
  ms2 <- summary(a2)[[1]][["Mean Sq"]]
  MSR <- ms2[1]; MSC <- ms2[2]; MSE <- ms2[3]

  icc21 <- (MSR - MSE) / (MSR + (k - 1) * MSE + k * (MSC - MSE) / n)
  icc31 <- (MSR - MSE) / (MSR + (k - 1) * MSE)
  icc2k <- (MSR - MSE) / (MSR + (MSC - MSE) / n)
  icc3k <- (MSR - MSE) / MSR

One-way: aov(score ~ subject) -> MSB, MSW

a1  <- aov(score ~ subject_f, data = dat)
  ms1 <- summary(a1)[[1]][["Mean Sq"]]
  MSB <- ms1[1]; MSW <- ms1[2]
  icc11 <- (MSB - MSW) / (MSB + (k - 1) * MSW)

F-test for ICC(3,1): MSR/MSE against F(n-1, (n-1)(k-1))

f_stat <- MSR / MSE
  f_df1  <- n - 1
  f_df2  <- (n - 1) * (k - 1)
  f_p    <- pf(f_stat, f_df1, f_df2, lower.tail = FALSE)

Koo & Li (2016) bands: <0.5 poor, 0.5-0.75 moderate, 0.75-0.9 good, >0.9 excellent

koo_li <- function(x) {
    if (is.na(x)) return("not estimable")
    if (x < 0.5) "poor" else if (x < 0.75) "moderate" else if (x < 0.9) "good" else "excellent"
  }
  headline_band <- koo_li(icc21)

  icc_results_df <- data.frame(
    icc_type = c("ICC(1,1) — one-way random, single rater",
                 "ICC(2,1) — absolute agreement, single rater",
                 "ICC(3,1) — consistency, single rater",
                 "ICC(2,k) — absolute agreement, average of all raters",
                 "ICC(3,k) — consistency, average of all raters"),
    value = round(c(icc11, icc21, icc31, icc2k, icc3k), 3),
    interpretation = sapply(c(icc11, icc21, icc31, icc2k, icc3k), koo_li),
    use_when = c(
      sprintf("Each %s is rated by a DIFFERENT random set of raters(no shared panel).", subj_h),
      sprintf("Raters are a random sample and absolute %s values must match — the default to report.", score_h),
      sprintf("These exact %d raters are the only ones of interest; systematic differences between them are forgiven.", k),
      sprintf("Decisions rest on the AVERAGE %s of all %d raters, and absolute values matter.", score_h, k),
      sprintf("Decisions rest on the average of these exact %d raters; only relative ordering matters.", k)
    ),
    stringsAsFactors = FALSE
  )

Step 6: Variance components (from the two-way mean squares)

var_subject <- max(0, (MSR - MSE) / k)
  var_rater   <- max(0, (MSC - MSE) / n)
  var_error   <- max(0, MSE)
  var_total   <- var_subject + var_rater + var_error
  if (var_total <= 0) var_total <- 1
  variance_df <- data.frame(
    source = c("Between subjects", "Between raters", "Residual"),
    variance_pct = round(100 * c(var_subject, var_rater, var_error) / var_total, 1),
    stringsAsFactors = FALSE
  )

Step 7: Rater means (systematic bias) on the complete-crossing data

rm_agg <- aggregate(score ~ rater_f, data = dat, FUN = mean)
  rater_means_df <- data.frame(
    rater_name = as.character(rm_agg$rater_f),
    mean_score = round(rm_agg$score, 2),
    stringsAsFactors = FALSE
  )

NA-safe extremes (never which.max over possibly-NA vectors)

ok_idx <- which(!is.na(rater_means_df$mean_score))
  hi_idx <- ok_idx[which.max(rater_means_df$mean_score[ok_idx])]
  lo_idx <- ok_idx[which.min(rater_means_df$mean_score[ok_idx])]
  rater_spread <- rater_means_df$mean_score[hi_idx] - rater_means_df$mean_score[lo_idx]

Step 8: Subject-level agreement — first two raters, one point per subject

d1 <- dat[dat$rater == rater_levels[1], c("subject", "score")]
  d2 <- dat[dat$rater == rater_levels[2], c("subject", "score")]
  names(d1)[2] <- "rater_1_score"
  names(d2)[2] <- "rater_2_score"
  merged <- merge(d1, d2, by = "subject")
  merged <- merged[order(merged$rater_1_score), , drop = FALSE]
  set.seed(42)
  if (nrow(merged) > 1000) merged <- merged[sample(nrow(merged), 1000), , drop = FALSE]
  agreement_df <- data.frame(
    rater_1_score = round(merged$rater_1_score, 3),
    rater_2_score = round(merged$rater_2_score, 3),
    stringsAsFactors = FALSE
  )
  agreement_r <- if (nrow(agreement_df) >= 3)
    suppressWarnings(cor(agreement_df$rater_1_score, agreement_df$rater_2_score))
  else NA_real_

  icc <- list(icc11 = icc11, icc21 = icc21, icc31 = icc31,
              icc2k = icc2k, icc3k = icc3k)

  metrics <- list(
    `Subjects`     = n,
    `Raters`       = k,
    `Ratings Used` = final_rows,
    `ICC(2,1)`     = round(icc21, 3),
    `Reliability`  = headline_band,
    `F-test p`     = signif(f_p, 3)
  )

  json_output <- list(
    answer = paste0(
      "Intraclass correlation from ", format(final_rows, big.mark = ","),
      " ratings of ", n, " ", subj_h, " values by ", k, " raters(", rater_h,
      "): ICC(2,1) = ", round(icc21, 3), " — ", headline_band,
      " absolute agreement for a single rater(Koo & Li). Consistency ICC(3,1) = ",
      round(icc31, 3), "; average-measure ICC(2,k) = ", round(icc2k, 3),
      ". The F-test for subject discrimination is ",
      ifelse(is.na(f_p), "not estimable",
             ifelse(f_p < 0.05, paste0("significant(p = ", signif(f_p, 3), ")"),
                    paste0("not significant(p = ", signif(f_p, 3), ")"))),
      ". Between-", subj_h, " differences explain ", variance_df$variance_pct[1],
      "% of the variance in ", score_h, "."
    ),
    cards = lapply(
      c("tldr", "overview", "preprocessing", "icc_results",
        "rater_means", "subject_agreement", "variance_breakdown"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = final_rows,
    rows_removed = rows_removed, n_invalid = n_invalid,
    subj_h = subj_h, rater_h = rater_h, score_h = score_h,
    n_subjects = n, k_raters = k, rater_levels = rater_levels,
    n_duplicates = n_duplicates, n_incomplete = n_incomplete,
    incomplete_subjects = incomplete_subjects,
    icc = icc, f_stat = f_stat, f_df1 = f_df1, f_df2 = f_df2, f_p = f_p,
    icc_results_df = icc_results_df,
    rater_means_df = rater_means_df,
    rater_hi = rater_means_df$rater_name[hi_idx],
    rater_lo = rater_means_df$rater_name[lo_idx],
    rater_spread = rater_spread,
    agreement_df = agreement_df, agreement_r = agreement_r,
    variance_df = variance_df,
    headline_band = headline_band,
    metrics = metrics, json_output = json_output
  )
}

This is the exact R code executed to produce the analysis above. Run it with the same dataset to reproduce these results.

Your data has more stories to tell. Run any analysis on your own data — 60+ validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No Signup Sign Up Free
Something wrong? Report it and we'll fix it — or get your credits back
Download PDF · Print · Share

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing