---
title: "McNemar or chi-square: which test do your paired yes/no data need?"
subtitle: "Paired proportions, decided. Companion analysis for the MCP Analytics video"
author: "MCP Analytics"
date: "2026-08-15"
output:
  html_document:
    theme: flatly
    toc: true
    toc_float: true
    code_folding: show
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
```

## What McNemar's test is for

McNemar's test answers one question about **paired yes/no data**: when the same
subjects are judged twice, does one judging say yes more often than the other?
Two reviewers on the same applications, two diagnostic tests on the same patients,
one model before and after retraining on the same cases: any time each row of your
data carries two binary verdicts about the same thing, this is the paired-proportions
question.

It is a test of **marginal homogeneity**, not of agreement. That distinction is the
whole reason this method needs a video: two judges can agree on most cases and still
be systematically biased in the cases where they differ. Agreement statistics look at
how often the judges match. McNemar looks only at the disagreements, and asks whether
they lean one way.

### Common applications

| Field | The question McNemar answers |
|---|---|
| Diagnostic testing | Do two tests, run on the same patients, differ in positive rate? (Trajman & Luiz 2008: compare sensitivities among the diseased) |
| Machine learning | Do two classifiers, scored on the same test set, differ in error rate? |
| Review pipelines | Do two reviewers approve at different rates on the same queue? |
| Clinical trials | Did a binary outcome change from before to after treatment in the same subjects? |
| Matched-pair A/B | Do matched units respond differently under two conditions? |

## What you need as input

One table, one row per subject or case, with **two binary verdicts per row**: the
same cases judged by both sources. That pairing is not a detail. It is the design
fact that decides which test is legal.

- **Verdicts must be dichotomous** (approve/deny, positive/negative, error/correct).
  More than two categories needs Stuart-Maxwell; more than two judgings of the same
  subjects needs Cochran's Q.
- **Power lives in the disagreements.** When using the McNemar test, "it is not
  the total sample size that determines power, but the total number of discordant
  pairs" (Pembury Smith & Ruxton 2020). Two hundred cases with six disagreements is a weaker study than
  sixty cases with thirty.

## What you get as output

The paired verdicts collapse into a 2x2 table with named cells: both-yes, both-no,
and the two kinds of disagreement. Write the disagreements as `b` (first judge yes,
second no) and `c` (first no, second yes). The test statistic uses **only b and c**:

$$\chi^2 = \frac{(b - c)^2}{b + c}$$

The concordant cells never enter the statistic. That is the mechanism behind the
video's central fact: agreement can be high while the test is decisive, because they
are computed from different cells.

One honesty requirement the literature is blunt about: **name your variant**. The
same 2x2 table yields different p-values under the uncorrected chi-square, the
Edwards continuity correction, the exact binomial, and the mid-p test, and software
defaults diverge three ways (R corrects by default, statsmodels is exact by default,
SPSS switches to the exact form at small
discordant counts, customarily below 25). Fagerland, Lydersen & Laake
(2013) compared them: their recommendation is the mid-p (or the exact unconditional
test), with the uncorrected asymptotic acceptable when small violations of the
nominal level are tolerable. Of the exact conditional and continuity-corrected
tests jointly they write that they "did not perform well for any of the considered
scenarios", and of the corrected version specifically, "we do not recommend that
it is used."

## The worked example

### The data and its provenance

Two reviewers decide approve or deny on the **same 200 applications**. The dataset
is synthetic and fully reproducible: generated by `mcnemar_example.py` (seed 42)
with cell counts fixed by design, because we want the ground truth visible: 120
both-approve, 50 both-deny, 6 approved only by reviewer A, 24 approved only by
reviewer B.

```{r data}
d <- read.csv("cases.csv")
head(d)
tab <- table(A = d$reviewer_a, B = d$reviewer_b)
tab
```

### First look: the surfaces a naive check reads say "fine"

```{r agreement}
agreement <- (tab["approve", "approve"] + tab["deny", "deny"]) / sum(tab)
pa <- sum(tab["approve", ]) / sum(tab)   # reviewer A approval rate
pb <- sum(tab[, "approve"]) / sum(tab)   # reviewer B approval rate
pe <- pa * pb + (1 - pa) * (1 - pb)
kappa <- (agreement - pe) / (1 - pe)
round(c(agreement = agreement, kappa = kappa, rate_A = pa, rate_B = pb), 3)
```

Agreement `r round(agreement, 2)` and kappa `r round(kappa, 2)`: numbers most
dashboards would wave through. But the approval rates already hint at the story:
reviewer B approves `r round(100*(pb - pa))` percentage points more of the same
queue.

### The disagreements are the evidence

Thirty cases got different verdicts, and they are lopsided: reviewer B approved
`r tab["deny", "approve"]` cases that A denied, while A approved only
`r tab["approve", "deny"]` cases that B denied. McNemar's test asks whether a
24-to-6 split could be chance if neither reviewer were systematically more
generous.

```{r mcnemar}
mcnemar.test(tab, correct = FALSE)   # the evidence-preferred headline
mcnemar.test(tab)                    # R's default applies the Edwards correction
b <- tab["approve", "deny"]; c <- tab["deny", "approve"]
c(exact_p = 2 * pbinom(min(b, c), b + c, 0.5),
  mid_p  = 2 * pbinom(min(b, c), b + c, 0.5) - dbinom(min(b, c), b + c, 0.5))
```

Every variant agrees the split is real (all p < 0.002). We headline the
**uncorrected** statistic, `r round((b-c)^2/(b+c), 1)` with p =
`r signif(mcnemar.test(tab, correct = FALSE)$p.value, 3)`, per Fagerland et al.;
the corrected value appears only so a reader reconciling against R's default output
knows why their number differs. With `r b + c` discordant pairs, every switch rule
in circulation (including the customary below-25 exact-switch rule) agrees the asymptotic form is
legitimate here.

### What the verdict means, and what it does not

The test rejects **marginal homogeneity**: reviewer B genuinely approves at a higher
rate than reviewer A on the same queue. It does not say the reviewers disagree
often (they agree on 85% of cases), and it does not measure agreement (that is
kappa's job; SAS's own documentation states the null as equality of the discordant
probabilities). High agreement and a decisive McNemar coexist without contradiction
because they are computed from different cells of the same table.

The actionable reading mirrors the ICC lesson's calibration experiment: this is a
**calibration finding, not a competence finding**. The reviewers rank cases
similarly; one of them holds a looser bar. The remedy is aligning the bar, not
retraining a reviewer.

### The classic mistake this chooser exists to prevent

The tempting error is the ordinary chi-square test of independence on the same
2x2 table. It is illegal here because the two verdicts on a case are not
independent samples: they are the same case, twice. Pembury Smith & Ruxton (2020)
document published examples of exactly this misuse, and their survey found papers
rarely even state which McNemar variant they used (none of their surveyed 50 used
the recommended mid-p). The chooser rule:

1. **Same subjects judged twice?** Paired: McNemar family. Independent groups:
   chi-square test of independence.
2. **Two categories?** Yes: McNemar. More categories: Stuart-Maxwell. More than two
   judgings: Cochran's Q.
3. **How many disagreements?** Below about 10 discordant pairs, warn rather than
   test (below b + c = 6 the exact p cannot reach 0.05 at all); otherwise report
   the uncorrected or mid-p variant, and always name which.

## Honest limitations

- **Only the discordant pairs carry information.** Our n = 200 sounds comfortable,
  but the test effectively runs on 30 cases. Design studies for disagreements, not
  for headcount.
- **Direction, not magnitude.** McNemar says the marginal rates differ; the effect
  size lives in the marginals themselves (63% vs 72% approval) and their difference,
  which should be reported alongside p.
- **Variant ambiguity is a real reproducibility hazard**: the same table yields
  p = 0.001015 (uncorrected), 0.001911 (corrected), 0.001431 (exact), or 0.000878
  (mid-p). Name the variant, always.
- **Not an agreement statistic.** Report kappa beside it when agreement is the
  question; they answer different questions and can disagree loudly.

## References

- McNemar, Q. (1947). Note on the sampling error of the difference between
  correlated proportions or percentages. *Psychometrika*, 12(2), 153-157.
- Edwards, A.L. (1948). Note on the "correction for continuity" in testing the
  significance of the difference between correlated proportions. *Psychometrika*,
  13(3), 185-187.
- Fagerland, M.W., Lydersen, S. & Laake, P. (2013). The McNemar test for binary
  matched-pairs data: mid-p and asymptotic are better than exact conditional.
  *BMC Medical Research Methodology*, 13, 91.
- Lancaster, H.O. (1961). Significance tests in discrete distributions. *JASA*,
  56(294), 223-234.
- Pembury Smith, M.Q.R. & Ruxton, G.D. (2020). Effective use of the McNemar test.
  *Behavioral Ecology and Sociobiology*, 74, 133.
- Trajman, A. & Luiz, R.R. (2008). McNemar chi-squared test revisited: comparing
  sensitivity and specificity of diagnostic examinations. *Scandinavian Journal of
  Clinical and Laboratory Investigation*, 68(1), 77-80.

## Session

```{r session}
sessionInfo()$R.version$version.string
```