# McNemar's test: agent brief

> **Audience: an agent deciding whether to run this analysis on a dataset.** Not marketing
> copy. Terse, complete, and honest about failure. If you are asked "should I run McNemar
> here", you should be able to answer from this file alone, **including saying no**.
>
> Tool: `standard_mcnemar` · free page: `/free/standard_mcnemar` ·
> worked example: `/worked-examples/mcnemar` · lesson: https://youtu.be/VEoq8mBLtkc

---

## 1. What it answers

**Did the yes/no rate change, when the same cases were judged twice.**

Two reviewers on one queue. One model before and after retraining. A diagnostic against a
gold standard. The distinguishing feature is **pairing**: every case appears under both
conditions, so the two results are not independent samples.

**The counter-intuitive part, and the whole reason the test exists:** the cases where both
judges agree carry **no information about which is stricter**. Only the disagreements do.
McNemar throws the agreements away on purpose.

**Questions it is mistaken for:**

| Actually asked | Right tool |
|---|---|
| *Are these two independent groups different?* (different cases each side) | `standard_ab_test` or a chi-square test of independence |
| *How much do the two raters agree?* | agreement and kappa. **Different question**, and this tool reports both beside the test so you can see they disagree |
| *Is the score the same?* (numeric, not yes/no) | `standard_icc` for reliability, `standard_group_comparison` for means |
| *Did the metric move after a launch?* (no pairing) | `standard_event_impact` |
| *Which of many variants is best?* | `standard_multiple_comparisons` |

**Chi-square is the specific wrong answer here.** It assumes independent samples. Paired
data breaks that assumption, and the test will happily return a number anyway.

## 2. When it applies, and when it does not

**Apply it when: the same cases were assessed twice, the outcome is binary, and you want to
know whether one condition says yes more often.**

**Do not apply it when:**

- **The two sets are different cases.** Then they are independent samples and this is the
  wrong test.
- **The outcome has more than two levels.** McNemar is 2x2. Use its extensions
  (Stuart-Maxwell, Bhapkar) or collapse deliberately.
- **You only have the marginal totals.** "A approved 126, B approved 144" cannot be tested:
  the discordant split is gone and it is the entire evidence base.
- **Very few disagreements.** With `b + c` small the asymptotic test is unreliable and the
  exact test must carry it. The tool handles this, but you should know it is happening.

## 3. What the data must look like

One row per **case**, with both decisions on it.

| key | type | meaning |
|---|---|---|
| `case` | any id | the thing judged twice |
| `rater_a` / condition 1 | binary | first decision |
| `rater_b` / condition 2 | binary | second decision |

**Row-level, never a 2x2 summary.** Aggregating to counts is the one transformation that
destroys the analysis, because the discordant cells are what the test reads.

**Sampling is dangerous here** in a way it is not elsewhere: sampling rows deletes
discordant pairs, which are rare and load-bearing. The tool declares `spec_json.sampling
mode=cap` for exactly this reason.

## 4. What it returns, and how to read each piece

| Output | Read it as | The trap |
|---|---|---|
| **McNemar results, all variants named** | uncorrected, continuity-corrected, exact and mid-p, each labelled | **four p-values, and they differ.** Reporting "the p-value" without naming the variant is the core abuse this tool was built to prevent |
| **The disagreements** | the b and c cells, which carry all the evidence | the concordant cells are excluded by design, not by oversight. A large sample with few disagreements is a small test |
| **Marginal rates** | the effect size: how much the yes-rate moved | a p-value says whether, never how much. Report this beside it |
| **Agreement and kappa** | how often the two judges matched at all | **deliberately shown as a contrast.** High agreement and a significant McNemar coexist happily: they answer different questions |
| **Exact-test fallback** | used when `b + c` is small | if it engaged, the asymptotic result was not trustworthy. That is a fact about your data |
| **AI insight** | plain-language verdict | reads the numbers it was given. It cannot know your cases were not truly paired |

## 5. How it fails

**The headline failure is reporting an unnamed p-value.** Four variants, four numbers, all
correct, and they support different write-ups. Fagerland's recommendation is that the
**uncorrected** test is the headline for most applied work, with the exact test as
sensitivity, which is why this tool names every variant rather than silently picking one.

**Failure modes that yield a plausible wrong answer rather than an error:**

- **Running chi-square on paired data.** It returns a number and the number is wrong.
- **Reading high agreement as "no change".** Agreement 85% with a significant McNemar is
  exactly the case in the worked example: they agreed on 170 of 200 and one was still
  materially stricter.
- **Sampling the rows**, which deletes discordant pairs and shrinks the evidence base
  invisibly.
- **Quoting the p-value without the marginal rates**, so nobody learns how big the shift was.
- **Assuming pairing that is not real.** If "the same case" was actually re-drawn or
  re-queued, the pairing is fictional and the test is unfounded. Nothing in the output can
  detect this.

## 6. Verified numbers you may cite

200 paired approve/deny decisions (seed 42). Cells: both-approve 120, both-deny 50,
**A-only 6, B-only 24**.

| quantity | value |
|---|---|
| agreement | 0.850 |
| kappa | 0.661 |
| chi-square, uncorrected (headline) | 10.8, p = 0.001015 |
| continuity-corrected | 9.6333, p = 0.001911 |
| exact | p = 0.001431 |
| mid-p | p = 0.000878 |
| the direction | **24 vs 6** |

**Agreement is 85% and the test is still highly significant.** Those two facts sit in the
same dataset and they are not in tension; they answer different questions. That is the
lesson.

**Do not say** the concordant cells were "ignored" as a limitation. They are excluded
because they carry no information about direction. That is the design.

**Verified four ways:** hand math in Python, R's `mcnemar.test` and `binom.test`, an
independent exact-fraction recompute sharing no code (19 of 19 numbers confirmed, including
the `b+c` exact-test boundary), and an adversarial read of the claims against primary
literature (six citations correct, three precision nits found and fixed).

**Live-tool reproduction: RUN AND MATCHED.** `standard_mcnemar` through the free-analyze
door, run `free_8gnp33EZuCoORwir_bd1f14896343` -> `rpt_NGMxryJlsnhqdhyVH6x8mgVQ79qErl9G`.
All variants match, including the 24-vs-6 direction, agreement and kappa.

## 7. Where everything is

| | |
|---|---|
| Tool | `standard_mcnemar` |
| Free page | https://mcpanalytics.ai/free/standard_mcnemar |
| Worked example | https://mcpanalytics.ai/worked-examples/mcnemar |
| Notebook | `/worked-examples/files/mcnemar.Rmd` |
| Validation record | `lattice/v2/refs/LAT-2246-edu-videos-stats-choosers/worked-example-mcnemar/VALIDATION.md` |
| Our tool's own run | `rpt_NGMxryJlsnhqdhyVH6x8mgVQ79qErl9G` (free-analyze, 2026-08-15, all variants matched) |

## 8. What building this caught, on the record

The first two acceptance runs computed on **50 rows instead of 200**, which exposed a
platform bug: the dataset door had been storing only the first 50 rows of *every* dataset
since 2026-08-11 (LAT-2259). Fixed and deployed before the accepted run.

Worth knowing for two reasons. The numbers above post-date that fix. And a truncated dataset
produces a *plausible* answer, not an error, so nothing but a contract with known-correct
expected values would have caught it.

## 9. Routing shortcut

```
Were the SAME cases judged twice, with a yes/no outcome?
├── no, different cases each side → NOT this. Independent samples: standard_ab_test.
├── outcome is numeric            → standard_icc (reliability) or group comparison (means)
├── more than two outcome levels  → NOT this. Stuart-Maxwell / Bhapkar, or collapse.
└── yes
    ├── only marginal totals, no row-level pairs? → cannot run. The discordant split is gone.
    ├── very few disagreements (b + c small)?     → run it, and report the EXACT variant
    └── report: the variant you chose BY NAME, its p-value, the marginal rates as the
        effect size, and agreement/kappa as the contrast. Never "the p-value" alone.
```