McNemar or chi-square: which test do your paired yes/no data need?

What McNemar’s test is for

McNemar’s test answers one question about paired yes/no data: when the same subjects are judged twice, does one judging say yes more often than the other? Two reviewers on the same applications, two diagnostic tests on the same patients, one model before and after retraining on the same cases: any time each row of your data carries two binary verdicts about the same thing, this is the paired-proportions question.

It is a test of marginal homogeneity, not of agreement. That distinction is the whole reason this method needs a video: two judges can agree on most cases and still be systematically biased in the cases where they differ. Agreement statistics look at how often the judges match. McNemar looks only at the disagreements, and asks whether they lean one way.

Common applications

Field	The question McNemar answers
Diagnostic testing	Do two tests, run on the same patients, differ in positive rate? (Trajman & Luiz 2008: compare sensitivities among the diseased)
Machine learning	Do two classifiers, scored on the same test set, differ in error rate?
Review pipelines	Do two reviewers approve at different rates on the same queue?
Clinical trials	Did a binary outcome change from before to after treatment in the same subjects?
Matched-pair A/B	Do matched units respond differently under two conditions?

What you need as input

One table, one row per subject or case, with two binary verdicts per row: the same cases judged by both sources. That pairing is not a detail. It is the design fact that decides which test is legal.

Verdicts must be dichotomous (approve/deny, positive/negative, error/correct). More than two categories needs Stuart-Maxwell; more than two judgings of the same subjects needs Cochran’s Q.
Power lives in the disagreements. When using the McNemar test, “it is not the total sample size that determines power, but the total number of discordant pairs” (Pembury Smith & Ruxton 2020). Two hundred cases with six disagreements is a weaker study than sixty cases with thirty.

What you get as output

The paired verdicts collapse into a 2x2 table with named cells: both-yes, both-no, and the two kinds of disagreement. Write the disagreements as b (first judge yes, second no) and c (first no, second yes). The test statistic uses only b and c:

\[\chi^2 = \frac{(b - c)^2}{b + c}\]

The concordant cells never enter the statistic. That is the mechanism behind the video’s central fact: agreement can be high while the test is decisive, because they are computed from different cells.

One honesty requirement the literature is blunt about: name your variant. The same 2x2 table yields different p-values under the uncorrected chi-square, the Edwards continuity correction, the exact binomial, and the mid-p test, and software defaults diverge three ways (R corrects by default, statsmodels is exact by default, SPSS switches to the exact form at small discordant counts, customarily below 25). Fagerland, Lydersen & Laake (2013) compared them: their recommendation is the mid-p (or the exact unconditional test), with the uncorrected asymptotic acceptable when small violations of the nominal level are tolerable. Of the exact conditional and continuity-corrected tests jointly they write that they “did not perform well for any of the considered scenarios”, and of the corrected version specifically, “we do not recommend that it is used.”

The worked example

The data and its provenance

Two reviewers decide approve or deny on the same 200 applications. The dataset is synthetic and fully reproducible: generated by mcnemar_example.py (seed 42) with cell counts fixed by design, because we want the ground truth visible: 120 both-approve, 50 both-deny, 6 approved only by reviewer A, 24 approved only by reviewer B.

d <- read.csv("cases.csv")
head(d)

##    case_id reviewer_a reviewer_b
## 1 case_001    approve    approve
## 2 case_002       deny    approve
## 3 case_003    approve    approve
## 4 case_004       deny    approve
## 5 case_005    approve    approve
## 6 case_006       deny       deny

tab <- table(A = d$reviewer_a, B = d$reviewer_b)
tab

##          B
## A         approve deny
##   approve     120    6
##   deny         24   50

First look: the surfaces a naive check reads say “fine”

agreement <- (tab["approve", "approve"] + tab["deny", "deny"]) / sum(tab)
pa <- sum(tab["approve", ]) / sum(tab)   # reviewer A approval rate
pb <- sum(tab[, "approve"]) / sum(tab)   # reviewer B approval rate
pe <- pa * pb + (1 - pa) * (1 - pb)
kappa <- (agreement - pe) / (1 - pe)
round(c(agreement = agreement, kappa = kappa, rate_A = pa, rate_B = pb), 3)

## agreement     kappa    rate_A    rate_B 
##     0.850     0.661     0.630     0.720

Agreement 0.85 and kappa 0.66: numbers most dashboards would wave through. But the approval rates already hint at the story: reviewer B approves 9 percentage points more of the same queue.

The disagreements are the evidence

Thirty cases got different verdicts, and they are lopsided: reviewer B approved 24 cases that A denied, while A approved only 6 cases that B denied. McNemar’s test asks whether a 24-to-6 split could be chance if neither reviewer were systematically more generous.

mcnemar.test(tab, correct = FALSE)   # the evidence-preferred headline

## 
##  McNemar's Chi-squared test
## 
## data:  tab
## McNemar's chi-squared = 10.8, df = 1, p-value = 0.001015

mcnemar.test(tab)                    # R's default applies the Edwards correction

## 
##  McNemar's Chi-squared test with continuity correction
## 
## data:  tab
## McNemar's chi-squared = 9.6333, df = 1, p-value = 0.001911

b <- tab["approve", "deny"]; c <- tab["deny", "approve"]
c(exact_p = 2 * pbinom(min(b, c), b + c, 0.5),
  mid_p  = 2 * pbinom(min(b, c), b + c, 0.5) - dbinom(min(b, c), b + c, 0.5))

##      exact_p        mid_p 
## 0.0014309064 0.0008779103

Every variant agrees the split is real (all p < 0.002). We headline the uncorrected statistic, 10.8 with p = 0.00102, per Fagerland et al.; the corrected value appears only so a reader reconciling against R’s default output knows why their number differs. With 30 discordant pairs, every switch rule in circulation (including the customary below-25 exact-switch rule) agrees the asymptotic form is legitimate here.

What the verdict means, and what it does not

The test rejects marginal homogeneity: reviewer B genuinely approves at a higher rate than reviewer A on the same queue. It does not say the reviewers disagree often (they agree on 85% of cases), and it does not measure agreement (that is kappa’s job; SAS’s own documentation states the null as equality of the discordant probabilities). High agreement and a decisive McNemar coexist without contradiction because they are computed from different cells of the same table.

The actionable reading mirrors the ICC lesson’s calibration experiment: this is a calibration finding, not a competence finding. The reviewers rank cases similarly; one of them holds a looser bar. The remedy is aligning the bar, not retraining a reviewer.

The classic mistake this chooser exists to prevent

The tempting error is the ordinary chi-square test of independence on the same 2x2 table. It is illegal here because the two verdicts on a case are not independent samples: they are the same case, twice. Pembury Smith & Ruxton (2020) document published examples of exactly this misuse, and their survey found papers rarely even state which McNemar variant they used (none of their surveyed 50 used the recommended mid-p). The chooser rule:

Same subjects judged twice? Paired: McNemar family. Independent groups: chi-square test of independence.
Two categories? Yes: McNemar. More categories: Stuart-Maxwell. More than two judgings: Cochran’s Q.
How many disagreements? Below about 10 discordant pairs, warn rather than test (below b + c = 6 the exact p cannot reach 0.05 at all); otherwise report the uncorrected or mid-p variant, and always name which.

Honest limitations

Only the discordant pairs carry information. Our n = 200 sounds comfortable, but the test effectively runs on 30 cases. Design studies for disagreements, not for headcount.
Direction, not magnitude. McNemar says the marginal rates differ; the effect size lives in the marginals themselves (63% vs 72% approval) and their difference, which should be reported alongside p.
Variant ambiguity is a real reproducibility hazard: the same table yields p = 0.001015 (uncorrected), 0.001911 (corrected), 0.001431 (exact), or 0.000878 (mid-p). Name the variant, always.
Not an agreement statistic. Report kappa beside it when agreement is the question; they answer different questions and can disagree loudly.

References

McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153-157.
Edwards, A.L. (1948). Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. Psychometrika, 13(3), 185-187.
Fagerland, M.W., Lydersen, S. & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13, 91.
Lancaster, H.O. (1961). Significance tests in discrete distributions. JASA, 56(294), 223-234.
Pembury Smith, M.Q.R. & Ruxton, G.D. (2020). Effective use of the McNemar test. Behavioral Ecology and Sociobiology, 74, 133.
Trajman, A. & Luiz, R.R. (2008). McNemar chi-squared test revisited: comparing sensitivity and specificity of diagnostic examinations. Scandinavian Journal of Clinical and Laboratory Investigation, 68(1), 77-80.

Session

sessionInfo()$R.version$version.string

## [1] "R version 4.5.1 (2025-06-13)"