# Multiple comparisons (Holm, Bonferroni, Benjamini-Hochberg): agent brief

> **Audience: an agent deciding whether to run this analysis on a dataset.** Not marketing
> copy. Terse, complete, and honest about failure. If you are asked "should I correct for
> multiple comparisons here", you should be able to answer from this file alone, **including
> saying no**.
>
> Tool: `standard_multiple_comparisons` · free page: `/free/standard_multiple_comparisons` ·
> worked example: `/worked-examples/holm` · lesson: https://youtu.be/ZVGFnlRzzlo

---

## 1. What it answers

**Of the results that looked significant, how many survive the fact that you ran many
tests.**

Test one hypothesis at alpha = .05 and you accept a 1-in-20 chance of a false positive. Test
66, and you should *expect* about three even if nothing is going on. Correction decides which
of your findings survive that arithmetic.

**The distinguishing feature is that the analysis has already happened.** This tool takes a
list of p-values you already have and adjusts them. It does not run the underlying tests.

**Questions it is mistaken for:**

| Actually asked | Right tool |
|---|---|
| *Are these two groups different?* | run the test first: `standard_ab_test`, `standard_group_comparison` |
| *Which variables move together?* | `standard_correlation`, then bring its p-values here |
| *How many samples do I need?* | power analysis, not this |
| *Which of my 20 metrics moved?* | this, on those 20 p-values, and the answer will be fewer than you think |
| *Is my model overfitting?* | a different problem entirely |

## 2. When it applies, and when it does not

**Apply it when: you have a set of p-values from tests that were run together, and a decision
rides on which are real.**

**Do not apply it when:**

- **You ran one test.** There is nothing to correct.
- **The tests are not a family.** Correction assumes a defined set. Correcting across
  unrelated analyses because they happened the same week is arbitrary, and so is the answer.
- **You picked which tests to report after seeing them.** Correction cannot repair selective
  reporting; it can only account for a family you declare honestly.
- **You need effect sizes.** This adjusts p-values. A surviving test can still be trivial.

## 3. What the data must look like

A list of p-values, one row per test, ideally with a label.

| key | type | meaning |
|---|---|---|
| `test` / label | any | which comparison this p-value belongs to |
| `p_value` | numeric, 0-1 | the uncorrected p-value |

**The family has to be complete.** Feeding 20 of the 66 tests you ran produces a correction
for a family of 20, which is wrong and looks fine. The count `m` is the whole analysis.

## 4. What it returns, and how to read each piece

| Output | Read it as | The trap |
|---|---|---|
| **Adjusted results** | each test's corrected p-value under each method | the methods disagree by design. Report which one you used and why, before you see the counts |
| **What survives correction** | how many findings remain, per method | **this is the number people quote and the number they choose after the fact.** Pick the method first |
| **P-value distribution** | the shape of the whole family | a pile near zero is a real signal; a flat spread is what noise looks like |
| **AI insight** | plain-language verdict | reads the numbers it was given. It cannot know your family was truncated |

## 5. How it fails

**The headline failure is choosing the correction after seeing which one gives you the
answer you wanted.** Holm and BH are both defensible; picking between them by counting
survivors is not.

The two control different things, and the choice is a business decision, not a statistical
one:

- **Bonferroni and Holm** control the chance of **any** false positive. Use them when one
  wrong call is expensive.
- **Benjamini-Hochberg** controls the *proportion* of your positives that are false. Use it
  when you are screening and can tolerate some noise to avoid missing real effects.

**Failure modes that yield a plausible wrong answer rather than an error:**

- **Reporting "we corrected" without naming the method.** The word does no work: 43 or 54
  survive on the same data depending on which you mean.
- **Truncating the family.** Correcting 20 of 66 tests inflates every survivor.
- **Using Bonferroni when Holm is available.** Holm is uniformly at least as powerful and
  controls the same error rate, so plain Bonferroni costs findings for nothing.
- **Reading a surviving p-value as an important effect.** Survival is not size.
- **The BH definition trap:** BH's count is the **largest** *i* with p(i) <= i·alpha/m, not
  the first failure. Implementing the intuitive version silently under-counts, and on many
  datasets the two agree, so it will not be caught by spot-checking.

## 6. Verified numbers you may cite

UCI Wine Quality (red), 1,599 wines, 12 chemistry variables, **all 66 pairwise Pearson
correlations**, alpha = .05, m = 66.

| method | survive |
|---|---|
| uncorrected | **55** |
| Bonferroni | **43** |
| Holm | **43** |
| Benjamini-Hochberg | **54** |

Smallest p-value: 4.06e-220. `rows_used 66`, not sampled.

**Twelve of the 55 raw "findings" do not survive Holm.** That is the number worth saying out
loud: more than a fifth of what looked significant was the arithmetic of looking 66 times.

**Holm and Bonferroni coincide exactly here (43 = 43), and that is a property of this
dataset, not of the methods.** Holm gains only when a p-value lands in the band between the
flat bar and the step-down bar. Here the first test Holm would reject beyond Bonferroni is
rank 44 at p = 2.213860e-03 against a bar of 2.173913e-03, missing by a hair.

**Do not say** the tool implements plain Bonferroni. It computes **Holm-Bonferroni** and
**Benjamini-Hochberg** and names them correctly throughout. A registry line once called the
43 "Bonferroni"; the count is right for both methods *because they coincide on this data*, so
a wrong method label survived four review passes and could not have been caught from the
numbers.

**Verified three ways:** hand-rolled step-down loops in Python, R's own `p.adjust` for all
three methods (the check that matters, since it shares no code), and a live tool run.

**Live-tool reproduction: RUN AND MATCHED.** Through the free-analyze door, run
`free_QCNyPUBLjWsYwsA1_0a63065ed9ef` -> `rpt_Bh9CakqVreJ4CHA6xmEpYy8TWSpe7XhL`: 66 tests,
0 invalid, 55 uncorrected, Holm 43, BH 54.

**Tested rather than cited:** "Holm is uniformly at least as powerful as Bonferroni" was
checked on this corpus rather than quoted. Bonferroni rejected nothing Holm did not.
Confirmed.

## 7. Where everything is

| | |
|---|---|
| Tool | `standard_multiple_comparisons` |
| Free page | https://mcpanalytics.ai/free/standard_multiple_comparisons |
| Worked example | https://mcpanalytics.ai/worked-examples/holm |
| Notebook | `/worked-examples/files/holm.Rmd` |
| Validation record | `lattice/v2/refs/LAT-2297-facelift-holm/worked-example/VALIDATION.md` |
| Our tool's own run | `rpt_Bh9CakqVreJ4CHA6xmEpYy8TWSpe7XhL` (free-analyze, 2026-08-17) |

## 8. Routing shortcut

```
Do you have MORE THAN ONE p-value from tests run as a family?
├── one test            → nothing to correct
├── tests unrelated     → NOT a family. Correcting across them is arbitrary.
├── family incomplete   → get all of them first. m is the whole analysis, not what you kept.
└── yes
    ├── is ONE wrong call expensive?      → Holm (never plain Bonferroni: Holm dominates it)
    ├── are you screening for candidates? → Benjamini-Hochberg
    └── choose BEFORE looking at the counts, name the method in the write-up, and
        remember that surviving correction says nothing about effect size.
```