Fisher's Exact Test: A Method Guide for Small-Sample 2x2 Tables

A Simulated Example With Run Output

Question: When two groups have the same true rate, how often does each test report p below 0.05 anyway?

Inputs: The campaign table from Section 5 (invented). Then simulated tables with a typed success rate of 0.2 in both groups, with 10, 20 and 40 per group, 20000 tables each, fixed seed 2026. The rate and sample sizes were typed by us, and the no-difference setup is built in.

Method: Python with numpy and scipy.stats. Each table is tested with Fisher's exact test, the chi-square test without correction and the chi-square test with continuity correction. Tables with an empty outcome column (no successes at all) cannot be tested with chi-square, so they are skipped for all three tests and counted in the last column.

Result: On the campaign table the run gives an odds ratio of 2.025, a two-sided Fisher p-value of 0.213 and a one-sided p-value of 0.131, matching Section 5. The chi-square p-values are 0.17 without correction and 0.262 with it. In the simulation, Fisher's test rejected in 0.65%, 2.26% and 2.79% of tables at 10, 20 and 40 per group. Staying under 5% is expected for an exact test, by design. These rates are among the tables that could be tested. Chi-square without correction rejected in 3.00%, 4.95% and 5.10%, near 5% at 20 and 40 per group (the Monte Carlo noise is about 0.15 points) and below it at 10. Tables with no successes at all were skipped: 230 of 20000 at 10 per group (about 1.2%), 2 at 20 and none at 40. With continuity correction it rejected in 0.65%, 1.83% and 2.12%.

Limit: These results are about this setup only: one rate, three sample sizes, two equal groups. This is one rate only: at a rate near 0.5 the chi-square test can behave differently. Other rates or sizes can give other shares. The run does not say how often any test is misused or how any test behaves on real data. No confidence interval is computed, because this script does not compute an exact one.

import numpy as np
from scipy.stats import fisher_exact, chi2_contingency

# Part A: the page's invented campaign table
table = [[15, 25], [8, 27]]
odds, p_two = fisher_exact(table, alternative="two-sided")
_, p_greater = fisher_exact(table, alternative="greater")
p_chi_plain = chi2_contingency(table, correction=False)[1]
p_chi_corr = chi2_contingency(table, correction=True)[1]
print("Part A: invented campaign table [[15,25],[8,27]]")
print("sample odds ratio:", round(odds, 3))
print("Fisher two-sided p:", round(p_two, 3))
print("Fisher one-sided (campaign higher) p:", round(p_greater, 3))
print("chi-square p, no correction:", round(p_chi_plain, 3))
print("chi-square p, continuity correction:", round(p_chi_corr, 3))

# Part B: tables drawn with NO true difference (same typed rate in both groups)
rate = 0.2
reps = 20000
rng = np.random.default_rng(2026)
print()
print("Part B: no true difference, rate", rate, "in both groups,", reps, "tables each")
print("n per group | Fisher | chi-square | chi-square with correction | tables skipped")
for n in (10, 20, 40):
    a = rng.binomial(n, rate, size=reps)
    b = rng.binomial(n, rate, size=reps)
    rej = [0, 0, 0]
    used = 0
    skipped = 0
    for x, y in zip(a, b):
        t = [[x, n - x], [y, n - y]]
        if x + y == 0 or x + y == 2 * n:
            skipped += 1  # chi-square is undefined with an empty column
            continue
        used += 1
        if fisher_exact(t)[1] < 0.05:
            rej[0] += 1
        if chi2_contingency(t, correction=False)[1] < 0.05:
            rej[1] += 1
        if chi2_contingency(t, correction=True)[1] < 0.05:
            rej[2] += 1
    print(n, "|", *[f"{100 * r / used:.2f}%" for r in rej], "|", skipped)

# Part C: Fisher's test gives an exact p-value; no confidence interval is computed here.
print()
print("Part C: no confidence interval is computed in this script.")

Printed output from running the code above:

Part A: invented campaign table [[15,25],[8,27]]
sample odds ratio: 2.025
Fisher two-sided p: 0.213
Fisher one-sided (campaign higher) p: 0.131
chi-square p, no correction: 0.17
chi-square p, continuity correction: 0.262

Part B: no true difference, rate 0.2 in both groups, 20000 tables each
n per group | Fisher | chi-square | chi-square with correction | tables skipped
10 | 0.65% 3.00% 0.65% | 230
20 | 2.26% 4.95% 1.83% | 2
40 | 2.79% 5.10% 2.12% | 0

Part C: no confidence interval is computed in this script.

1. Introduction

Fisher's Exact Test is a nonparametric test of independence for categorical data, most often applied to a 2x2 contingency table. It was developed by Sir Ronald A. Fisher and described in The Design of Experiments (1935), using the well-known example of a person claiming to tell whether milk or tea was poured into a cup first. The test enumerates the tables that share the observed row and column totals and sums the probabilities of those at least as extreme as the one observed.

Because that calculation does not rely on a large-sample approximation, the p-value is exact given the margins. This contrasts with the Pearson chi-square test, whose reference distribution is only an approximation and can be unreliable when expected cell counts are small.

Scope and Purpose

This paper is a method guide. Apart from one simulated example with its code and run output, it reports no original research or data analysis, and it makes no claim about how often the test is misused in practice. Its purpose is to explain:

1. What the test computes and the formula behind it.
2. When it is appropriate, given sample size, expected counts, table shape and the independence of observations.
3. How to interpret and report it, including tail choice and effect sizes.

2. Background

The Challenge of Small-Sample Categorical Analysis

When samples are large, Pearson's chi-square test gives a convenient approximation for testing independence in a contingency table. That approximation depends on each cell having a sufficiently large expected count, traditionally at least five. When expected counts are small, the chi-square p-value can be inaccurate. Sparse tables, such as a rare outcome observed in a small number of subjects, are exactly where an exact method is attractive.

Alternatives to Fisher's Test

Continuity correction: Yates' correction adjusts the chi-square statistic for the fact that counts are discrete. It is still an approximation, and it is often described as conservative.

Permutation tests: Randomization approaches build a null distribution by permuting the data. For a 2x2 table with fixed margins this reproduces Fisher's test.

Barnard's test: An unconditional exact test that does not fix both margins. Lydersen, Fagerland and Laake (2009) report that unconditional tests keep the significance level and are generally more powerful than Fisher's exact test for small to moderate samples. They add that the old objection, the computing cost, is now moot because software for these tests has been available for years, and that Fisher's test with a mid-p adjustment gives about the same results. Their recommendation is to use unconditional tests and the mid-p approach more often, and they say the traditional Fisher's exact test should practically never be used. This page still explains the traditional test because it is widely reported and is the building block for the mid-p version.

Mid-p correction: A variant that counts only half the probability of the observed table. It is less conservative than the standard exact p-value but does not guarantee the nominal Type I error rate.

Theoretical Foundation

Fisher's Exact Test uses the hypergeometric distribution. For a 2x2 table with cells labeled:

the probability of this exact configuration under the null hypothesis of independence, given the margins, is:

The p-value is the sum of P(table) over all tables with the same margins that are at least as extreme as the observed one, where "extreme" depends on the alternative hypothesis. Conditioning on the margins is what makes the p-value exact under the independence assumption.

3. Method

Approach of This Paper

This paper explains an established method from the standard statistical literature (see References), and it states no measured error rates, timings or business outcomes.

Data Considerations

Sample size: The test is valid at any sample size. It is most useful when counts are small. With very large samples it gives results close to the chi-square test and can be slower for larger tables.

Expected cell frequencies: Compute expected counts under independence (row total times column total divided by n). If any is below about 5, the chi-square approximation is questionable and Fisher's test is a natural alternative.

Table dimensions: The classic test is for 2x2 tables. The Freeman-Halton extension covers r x c tables, but the number of tables to enumerate grows quickly with table size and sample size.

Independence structure: Each observation should be an independent sampling unit. Pairing, matching, clustering and repeated measures on the same units break this assumption.

Hypothesis direction: Decide whether the question is directional (one-tailed) or not (two-tailed) before looking at the results.

Tools

The test is available in common software, for example fisher.test() in R and scipy.stats.fisher_exact() in Python. Effect-size helpers such as OddsRatio() and RelRisk() are available in the R package DescTools.

4. Key Points

# R example with effect sizes
library(DescTools)

# Pass your own four counts: the first group's two outcomes, then the second group's two outcomes
fisher_report <- function(n11, n12, n21, n22) {
  contingency_table <- matrix(c(n11, n12, n21, n22), nrow = 2, byrow = TRUE,
    dimnames = list(Group = c("Group 1", "Group 2"), Outcome = c("Outcome A", "Outcome B")))

  # Fisher's Exact Test
  fisher_result <- fisher.test(contingency_table)

  # Odds ratio with 95% CI
  odds_ratio <- OddsRatio(contingency_table, conf.level = 0.95)

  # Relative risk with 95% CI
  rel_risk <- RelRisk(contingency_table, conf.level = 0.95)

  # Report both significance and effect size
  cat("Fisher's Exact Test: p =", fisher_result$p.value, "\n")
  cat("Odds Ratio:", odds_ratio[1], "95% CI:", odds_ratio[2], "-", odds_ratio[3], "\n")
  cat("Relative Risk:", rel_risk[1], "95% CI:", rel_risk[2], "-", rel_risk[3], "\n")
}

5. Worked Example (Illustrative)

Illustrative example with made-up numbers. Suppose a campaign was shown to 40 people and 15 of them converted, while a comparison group of 35 people had 8 conversions. These figures are invented to show the mechanics and do not describe any real campaign.

Running Fisher's Exact Test on this table (for example with scipy.stats.fisher_exact) gives a sample odds ratio of (15 × 27) / (25 × 8) = 2.025, a two-tailed p-value of about 0.21, and a one-tailed (campaign higher) p-value of about 0.13. Suppose the team had committed to the one-tailed hypothesis before the campaign. Neither p-value would be below 0.05 here. The odds ratio of about 2 points to a possibly meaningful difference, but with groups this small the data cannot rule out chance, which is a reminder to read the effect size and its confidence interval alongside the p-value.

If the one-tailed test had been chosen only after noticing that the campaign group converted more often, the one-tailed p-value would not carry its nominal meaning.

6. Practical Guidance

The points below are general guidance drawn from the method's assumptions. They are not findings from any study.

7. Conclusion

Fisher's Exact Test gives an exact p-value for association in a 2x2 table by conditioning on the margins and using the hypergeometric distribution. It is a sensible choice when expected counts are small or an exact p-value is wanted, provided observations are independent.

Using it well is mostly a matter of procedure: check independence, fix the hypothesis and its direction in advance, report an effect size with its interval, and plan for the cost of larger tables. This paper explained the method and its assumptions. It did not measure how often any of these points matter in practice, and it should be read as a reference rather than as evidence about outcomes.