Holm-Bonferroni Method: Formula, Steps and When to Use It

The short answer

Take 66 correlation tests from 1,599 red wines. 55 of them are significant at 0.05. Holm keeps 43, exactly what Bonferroni keeps. Benjamini-Hochberg keeps 54, and it assumes the tests are independent or positively dependent. The 12 that Holm and Bonferroni drop are significant at 0.05 but did not clear the stricter bar, not proven chance results: a stricter bar can drop a true effect.

Have your own p-values? Run Bonferroni, Holm and Benjamini-Hochberg side by side in the free multiple comparisons tool.

For why a correction is needed at all, read the multiple comparisons problem.

Holm never keeps fewer than Bonferroni, and here it keeps the same. It gains only when a p-value lands between alpha / m and alpha / (m - k), where k is how many Bonferroni already keeps. On this data nothing lands there: the first test both methods drop, volatile acidity against total sulfur dioxide (r = 0.076), misses Holm's bar by about 2 percent. So the real choice on this data is strict against lenient, 43 against 54, not Holm against Bonferroni. Holm gives the same guarantee as Bonferroni: whichever of your null hypotheses are true, the chance that you declare even one of those significant stays at or below your alpha. It needs no assumption about how your tests relate to each other.

Our position: if you need that guarantee, use Holm and never plain Bonferroni. If you can live with a known share of false discoveries among your findings, use Benjamini-Hochberg instead. Do not mix the two in one family.

How the Holm thresholds climb

Holm sorts the p-values from smallest to largest and tests them in that order. The i-th smallest is compared with alpha / (m - i + 1), where m is the number of tests. So the first test faces alpha / 66, the second alpha / 65, and so on, one step easier each time. Here is the walk on the wine family, at alpha 0.05:

The bar rises from 0.000758 at rank 1, the same as Bonferroni's single cutoff, to 0.002174 at rank 44. Every one of the first 43 p-values sits far below its own bar. Rank 44 is the first to land above it, by a margin of about 0.00004, and that is where the walk ends: Holm retains rank 44 and everything ranked after it, even the tests that would pass an unadjusted 0.05. Only a p-value between Bonferroni's 0.000758 and Holm's 0.002174 at that rank could have been a gain, and none of the 66 falls in that gap. (The bars are arithmetic; the p-values come from the file described below.)

The data behind it

The 66 tests come from the Wine Quality data of Cortez, Cerdeira, Almeida, Matos and Reis (2009), UCI Machine Learning Repository, DOI 10.24432/C56S3T, licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). The 66 p-values are our computation from the file; the data are unchanged. Every red wine in it has eleven lab measurements and a sensory quality score. Pairing each of those twelve columns with every other gives 66 Pearson tests, all run at alpha 0.05 on the same 1,599 bottles.

Holm's last keeper is residual sugar against pH, with r = -0.086. Its raw p of 0.000607 becomes a Holm-adjusted 0.0146: the raw p times 24, the number of tests still in play at rank 43. That is a correlation you can detect with this many wines and could barely see on a scatter plot. A step-down correction rules on chance, not on size, so read r beside every kept pair.

Because the tests reuse the same bottles, and each column enters eleven pairs, their p-values move together. Holm's guarantee does not depend on that structure, so the 43 stand as they are. If nothing in the data were related, about 3.3 of 66 tests would still reach 0.05 by luck (66 x 0.05).

Holm against the other corrections

The family-wise rate is the probability of one or more false positives anywhere in the family. Holm holds it at alpha in the strong sense: under every mix of true and false nulls, not only when all are true. That is the whole reason to pick it over a false discovery rate method, whose promise is about the average share of mistakes in the kept set.

Reading a Holm result

• Each kept test inherits the family's guarantee. With alpha at 0.05, the probability that even one of the 43 is a false positive is at most 5 percent. Report each with its r and interval, because the step-down rule says nothing about magnitude.
• The stop is final. Once rank 44 misses, ranks 45 to 66 are retained without looking at them, even the ones under 0.05. Do not cherry-pick a later test back in.
• Retained is not refuted. A test after the stop simply lacked the evidence to clear a bar raised by the size of the family. Its interval is still worth showing.
• Fix the family before the walk. Dropping a test after seeing it shortens m and loosens every bar; adding a duplicate lengthens m and tightens them.
• Correlated tests cost Holm power, not validity of its promise. When tests move together strongly, a permutation-based step-down can keep more under the same guarantee.
• A clear primary outcome changes the design. If one hypothesis matters most, a fixed-sequence or gatekeeping order can beat a flat Holm family.

How to do it on your own data

1. Gather the unrounded p-value for every test in the family; rounding creates ties that scramble the order.
2. Sort ascending and number the ranks 1 to m.
3. Give rank i the bar alpha / (m - i + 1).
4. Walk from rank 1. Keep while the p-value is at or under its bar. At the first miss, stop and retain that test and all later ones.
5. For adjusted p-values: multiply p(i) by (m - i + 1), carry the running maximum forward, cap at 1, and compare every result with alpha. Same decisions as the walk.

The walk, written out, then the library call that does it:

import numpy as np
            from statsmodels.stats.multitest import multipletests

            p = np.asarray(p)
            order = np.argsort(p)
            m = len(p)
            keep = np.zeros(m, dtype=bool)
            for step, idx in enumerate(order):          # step 0 is rank 1
                if p[idx] <= 0.05 / (m - step):
                    keep[idx] = True
                else:
                    break                               # Holm stops at the first miss

            reject, p_holm, _, _ = multipletests(p, alpha=0.05, method="holm")
            assert (reject == keep).all()

An illustration, not a result. Six tests with these made-up p-values: 0.001, 0.009, 0.025, 0.040, 0.120, 0.350, at alpha 0.05. The bars are 0.0083, 0.0100, 0.0125, 0.0167, 0.0250, 0.0500. The first two pass (0.001 is at most 0.0083; 0.009 is at most 0.0100). The third, 0.025, is above 0.0125, so the walk stops and the other four are retained. The adjusted p-values are 0.006, 0.045, 0.100, 0.120, 0.240 and 0.350. Four raw p-values are below 0.05, and only two survive.

Planning sample size for Holm. Power each test as if it faced rank 1's bar, alpha / m, because the smallest effect you care about may be the first one tested. Later ranks can only be easier. If that sample is out of reach, shrink the confirmatory family to the questions that matter and move the rest into an exploratory set.

Run it on your own p-values

Paste or upload a column of p-values, up to 10,000 rows, and the free tool returns the Holm decisions and adjusted p-values next to Bonferroni and Benjamini-Hochberg, with the R code that produced them. The ready-check on the left says what the file needs.

Run the free correction on your file · Try the example file first

Questions analysts ask

Is Holm ever worse than Bonferroni?

No. Every hypothesis Bonferroni rejects, Holm rejects too, under the same family-wise guarantee. On the wine family the two tie at 43; elsewhere Holm can keep more. Bonferroni's one advantage is that it needs no sorting.

Does Holm need independent tests?

No. Its control of the family-wise error rate holds under any dependence. Strong positive correlation makes it more conservative than it needs to be, which is a loss of power, not a broken promise.

Why does Holm stop at the first miss instead of checking every test?

Because each bar is only valid if every earlier hypothesis was rejected. Once rank i misses, the argument that justified alpha / (m - i) for rank i + 1 no longer holds, so every later test is retained.

What if my p-values are rounded, or one is reported as less than 0.0001?

Rounding causes ties, which break the ordering; use the unrounded values when you have them. Treat a bound such as <0.0001 as the bound itself. That can only make a result harder to keep, never easier.

How many tests do I need before Holm and Bonferroni differ?

With under about ten tests the two bars barely separate. The gap widens as the family grows and as more tests pass early. For hundreds of tests, a false discovery rate method usually fits the question better.

Writing this up for publication? See statistical analysis for researchers. Citations and R code included.