Benjamini-Hochberg Procedure: Formula, Steps and When to Use It

The short answer

On a family of 66 wine correlations, Benjamini-Hochberg at a 5 percent false discovery rate (FDR) retains 54. The strict family-wise corrections, Holm and Bonferroni, retain 43, and 55 had p under 0.05 to begin with. So this procedure keeps 11 results the strict pair let go, and drops just one that cleared 0.05 unadjusted. Its promise rests on an assumption: the tests must be independent or positively dependent.

Have your own p-values? Run Bonferroni, Holm and Benjamini-Hochberg side by side in the free multiple comparisons tool.

The problem all three corrections answer: the multiple comparisons problem.

The thing being controlled is different. A family-wise method caps the probability that even one kept result is a false positive. Benjamini-Hochberg caps the expected proportion of false positives among the results you keep: at 5 percent, on average no more than about 1 in 20 of them. The wine tests reuse the same bottles and the same columns, so they are not independent, and whether their dependence is positive cannot be read off this page. Treat the 54 accordingly.

Our position: use it as a screen, when a later step will check the leads and a few dead ends are affordable. When each kept result will be reported or acted on directly, prefer Holm. Settle the rule and the rate before seeing the p-values, and use one rule per family.

The rank rule that controls the false discovery rate

Order the 66 p-values from smallest to largest and number them. Rank j gets its own critical value, (j / 66) x 0.05, a ramp that rises in equal steps from 0.00076 at rank 1 to 0.05 at rank 66. Then look for the highest rank whose p-value is at or under its ramp value. That rank and everything ranked before it are kept. The scan conventionally runs from the top of the list downward, which is why the method is called step-up.

The boundary on the wine family:

Rank 54 passes with 0.0389 against 0.04091. Rank 55 fails by a hair, 0.0428 against 0.04167, and no higher rank recovers. So the cut sits between 54 and 55. Rank 44 shows how much more room the ramp gives than the strict corrections: its p of 0.00221 is about fifteen times smaller than its ramp value of 0.03333, while it sits above the family-wise bars at that rank. (Ramp values are arithmetic; p-values are from the file below.)

The data behind it

Data credit: Cortez, Cerdeira, Almeida, Matos and Reis (2009), Wine Quality, UCI Machine Learning Repository, DOI 10.24432/C56S3T, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). The 66 p-values are our computation from the file; the data are unchanged. The red-wine table has 1,599 rows and 12 numeric columns, eleven chemistry readings and one quality score. Testing the Pearson correlation of each column against each other column yields 66 p-values at alpha 0.05.

The 11 extra results, ranks 44 to 54, all have correlations between 0.05 and 0.08 in magnitude. The last of them, free sulfur dioxide against sulphates (r = 0.052), has a Benjamini-Hochberg-adjusted p of 0.0475. Its neighbour at rank 55, free sulfur dioxide against quality (r = -0.051), adjusts to 0.0514 and falls outside. With 1,599 rows, associations this small reach significance easily, so the procedure is telling you which leads are worth checking, not which ones matter.

On dependence, be explicit. Each of the 12 columns feeds 11 of the pairs, and every pair uses the same 1,599 bottles. The original proof covers independent tests, and the guarantee is known to extend to positive dependence. Whether that holds for these 66 is not established here, so present the 54 as candidates for follow-up. If you cannot defend positive dependence, the Benjamini-Yekutieli variant divides every ramp value by the sum of 1/i for i = 1 to m and holds under any dependence, keeping fewer in exchange.

Benjamini-Hochberg and the other corrections

The top three rows protect against any error at all; the bottom two protect the average quality of the kept list. Choosing between them is choosing which mistake is expensive in your setting, a decision about consequences rather than a contest over who keeps the most.

How to read the 54

• The rate describes the list, not the item. At 5 percent, the expected share of false positives among the 54 is at most 5 percent, on average about 2.7 of them at most (54 x 0.05), and you cannot tell which. Under the assumption, the bound is alpha times the fraction of true nulls, so it may be lower.
• The list has to be fixed first. Every test that could be read as a finding belongs in the family before any p-value is seen. Editing the family afterwards voids the rate.
• Set the rate by what a false lead costs. Five percent is the common default. Ten or twenty percent yields more leads and more wasted follow-ups; decide in advance.
• Weaker evidence, different yardstick. The 11 extra results have larger p-values than anything Holm keeps. They are in because the ramp asks a looser question, so print the rate next to the count.
• Report magnitudes for all 54. Several are tiny correlations, and because they share rows they are not separate discoveries.
• Outside the cut is not a null. A test past rank 54 simply lacked the evidence for this rate on a family this size.

Running it yourself

1. Collect every p-value in the family, unrounded.
2. Rank them 1 to m, smallest first.
3. Compute each rank's ramp value, (j / m) x alpha.
4. Find the largest j whose p-value is at or under its ramp. Keep ranks 1 to j; if no rank passes, keep nothing.
5. Adjusted p-values, if you need them: p(j) x m / j, then a running minimum taken from the largest rank down, capped at 1. Keep where the adjusted value is at or under alpha. Same decisions as step 4.

The ramp, then the library call:

import numpy as np
            from statsmodels.stats.multitest import multipletests

            p = np.asarray(p)
            m = len(p)
            order = np.argsort(p)
            ramp = 0.05 * np.arange(1, m + 1) / m
            passing = np.nonzero(p[order] <= ramp)[0]
            k = passing.max() + 1 if passing.size else 0   # last rank under the ramp
            keep = np.zeros(m, dtype=bool)
            keep[order[:k]] = True

            reject, p_bh, _, _ = multipletests(p, alpha=0.05, method="fdr_bh")
            assert (reject == keep).all()

Planning a screen. The procedure lowers the price of testing many hypotheses but does not cancel it; each test still needs enough data to detect the effects you care about at its own level. Budget for the follow-up from the start, because the second look is what turns a screen into a finding.

Screen your own list

Give the free tool a column of p-values (up to 10,000 rows) and it returns which ones Benjamini-Hochberg keeps at your rate, the adjusted values, and the same list under Holm and Bonferroni for comparison, with R code. The ready-check on the left explains the input.

Run the free correction on your file · Try the example file first

Questions analysts ask

What does a 5 percent false discovery rate actually promise?

That, averaged over many studies of the same design, at most 5 percent of the results you keep are false positives, if the assumptions hold. It promises nothing about any one result, and nothing about the chance that at least one is wrong.

Does Benjamini-Hochberg need independent tests?

It needs independence or positive dependence among the tests. Shared rows, shared variables and metrics that rise and fall together are the usual cases to think about. If the assumption is doubtful, use Benjamini-Yekutieli, or Holm if you want a guarantee with no dependence condition.

How can a test above its own ramp value still be kept?

Because the procedure keeps everything up to the last rank that passes. A test in the middle of the list can sit above its own ramp and still be kept, provided some later rank is under its ramp.

How should I choose the rate?

By the cost of chasing a false lead. If a check is cheap, 10 percent may be reasonable; if each lead costs a study, use 5 percent or switch to a family-wise method. Fix it before looking.

What counts as the family?

All the tests whose results will be read together as one answer, decided before any p-value is seen. Unrelated questions can be separate families; many metrics read off one experiment are one.

When is Holm the better choice?

When every kept result will be quoted or acted on, so a single false positive is costly. Holm bounds the chance of any error and makes no dependence assumption.

Writing this up for publication? See statistical analysis for researchers. Citations and R code included.