Bonferroni Correction: Formula, Steps and When to Use It

The short answer

Divide 0.05 by the 66 pairings of a red-wine chemistry file and you get one cutoff, 0.000758, that every test must meet. Of the 66, 55 have p under 0.05 before any adjustment; 43 also get under the cutoff. Twelve fall in between. They are significant at 0.05 but did not clear the stricter bar, which is a statement about evidence, not a finding that they are chance.

Test your own list of p-values against Bonferroni's flat bar, with Holm and Benjamini-Hochberg beside it, in the free tool.

The idea behind all three: the multiple comparisons problem.

What you buy with that single number is a family-wise promise: across all 66, the probability of calling any true null significant is no more than 0.05. The proof is a union bound, so it asks nothing about whether the tests are related. What you pay is power, since each test is judged at one sixty-sixth of the usual level.

Our position: once the p-values are in hand, run Holm instead, because it can never keep fewer and carries the same promise; on this file it lands on the same 43. Reach for the single cutoff when it has to be written down before data exist, or when you are auditing a paper that reports only how many tests it ran. Screening work belongs to a false discovery rate method. Pick one rule per family.

How the Bonferroni threshold works

The rule has one moving part. Count the tests, m. Divide alpha by m. Any p-value at or below the quotient is significant after correction, and nothing else is. No sorting, no ranks, no stopping rule.

An equivalent way to report it: multiply each raw p by m, cap the product at 1, and compare that adjusted value with alpha. For the wine family, m = 66:

Residual sugar against pH is the closest call that survives: 0.0400 after multiplying, comfortably under 0.05. The next test up the list, volatile acidity against total sulfur dioxide, is nearly three times the cutoff, so the line between kept and dropped is easy to see by eye. (Adjusted values are 66 times the raw p from the file below.)

The data behind it

Source: Wine Quality, by Cortez, Cerdeira, Almeida, Matos and Reis (2009), UCI Machine Learning Repository, DOI 10.24432/C56S3T, under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). The 66 p-values are our computation from the file; the data are unchanged. It records 1,599 red wine samples, each with eleven physicochemical readings plus a quality grade. Correlating every column with every other, one Pearson test per pair, produces the family of 66.

These twelve sit between 0.000758 and 0.05, ranked 44 to 55 by p-value. All of them are significant at 0.05 and none meets the cutoff:

Look at the r column. The largest in magnitude is 0.076, so even if every one of these were real, each would explain well under 1 percent of the variance in its partner. That is the usual profile of what a strict cutoff removes from a big sample: small associations with modest evidence. Labelling them "not shown at this bar" is accurate; labelling them "noise" is not.

Every pair shares the same 1,599 bottles and every column appears in eleven pairs, so the 66 p-values are correlated with each other. Bonferroni's promise survives that. When the dependence is positive, the true family-wise rate ends up below 0.05, which is another way of saying the cutoff is stricter than it has to be. Under a global null, 66 tests at 0.05 would throw up about 3.3 significant results on average (66 x 0.05).

Bonferroni next to Holm, Hochberg and Benjamini-Hochberg

The first three columns of that table separate into two goals. Bonferroni, Holm and Hochberg bound the probability of a single mistake anywhere. Benjamini-Hochberg bounds the average proportion of mistakes among what you keep. Both are legitimate; they answer different questions. On these wines Bonferroni and Holm agree at 43, which is this dataset's outcome, not a property of the methods.

Using a Bonferroni result well

• What a kept test means. The probability that any of the 43 is a false positive is at most 5 percent, as a group. It is not a statement that each effect is large; give r and an interval for each.
• The power bill grows with m. Each test here is judged at 0.000758 instead of 0.05. A modest true effect that would pass alone can easily miss, so below the cutoff means unproven, not absent.
• Count m honestly and before looking. Remove a test after seeing its p and the cutoff loosens without justification; count one twice and it tightens for no reason.
• The cutoff does not fix the underlying tests. Wrong distributional assumptions or small samples in the base tests carry straight through.
• Separate questions can be separate families. Planned comparisons that each answer a distinct question may be corrected apart, if that split is decided in advance.
• At very large m, switch goals. With hundreds or thousands of tests alpha / m becomes tiny, and controlling the false discovery rate is usually the more useful target.

Doing it by hand or in code

1. Before results arrive, write down m, the number of tests in the family.
2. Compute alpha / m. Ten tests at 0.05 give 0.005; this file's 66 give 0.000758.
3. Keep each test whose raw p is at or below that number.
4. If you prefer adjusted p-values, report min(1, m x p) and compare each with alpha. Identical decisions.

The cutoff, then the library call that does the same thing:

import numpy as np
            from statsmodels.stats.multitest import multipletests

            p = np.asarray(p)
            cutoff = 0.05 / len(p)            # one bar for every test
            keep = p <= cutoff

            reject, p_bonf, _, _ = multipletests(p, alpha=0.05, method="bonferroni")
            assert (reject == keep).all()

Sizing a study for Bonferroni. Run the power calculation at alpha / m, not at alpha. That is the level every test will face. If the required sample is too large, trim the confirmatory family to the comparisons that actually drive a decision and treat the rest as exploratory.

Check your own family

Drop in a column of p-values (up to 10,000 rows) and the free tool shows the Bonferroni cutoff, each adjusted p-value and the keep or drop call, alongside Holm and Benjamini-Hochberg on the same list, plus the R code. The ready-check on the left lists the file format.

Run the free correction on your file · Try the example file first

Questions analysts ask

How do I calculate a Bonferroni correction?

Take your significance level and divide by the number of tests, alpha / m. A test passes if its raw p is no larger than that. Or multiply every p by m, cap at 1, and compare with alpha; the two give the same answer.

Is Bonferroni too conservative?

Of the common corrections it is the strictest: one cutoff for every test, with no use of how the p-values are ordered. Holm never keeps fewer, and with many tests or tests that move together Bonferroni can keep far fewer than the family-wise level allows. That strictness is the price of making no assumption.

Does Bonferroni assume the tests are independent?

No. Its guarantee follows from a union bound, which holds whatever the dependence. Correlation among the tests changes how much power you lose, not whether the 5 percent promise holds.

If Holm is never worse, why does anyone use Bonferroni?

Because the cutoff can be fixed in a protocol before a single p-value exists, and anyone can check it with a calculator. After the data are in, Holm gives at least as many results under the same promise, so it is the better default.

Bonferroni or a false discovery rate method?

They target different errors. Bonferroni caps the chance of any false positive in the family; Benjamini-Hochberg caps the expected proportion of false positives among the results it keeps. On the wine file that is 43 kept against 54, the 54 resting on Benjamini-Hochberg's dependence assumption.

Which tests can I apply it to?

Any family of valid p-values: t-tests, post-hoc ANOVA contrasts, correlation tests, chi-square tests, regression coefficients. It adjusts the decision cutoff only, so it cannot rescue a test whose own assumptions fail.