When the standard method isn't enough.

Worked example: an exact test on eight cups of tea

Each paper on this shelf explains a method with its assumptions. Here is one method run end to end, so you can see the kind of result the papers work toward.

The data. In The Design of Experiments (1935), Ronald Fisher described a lady who said she could taste whether milk or tea went into the cup first. The test: eight cups, four poured milk first and four tea first, served in random order. She is told there are four of each and must name the four milk-first cups. The data set is these eight cups.

The method. Fisher's exact test. It asks: if she were only guessing, how likely is a result at least this good? With four cups chosen out of eight, there are 70 equally likely ways to guess (8 choose 4). The p-value is the share of those 70 guesses that score as well as she did or better.

Assumptions. The cups are served in random order; she knows there are exactly four of each kind (so both row and column totals are fixed); and under "only guessing" every set of four is equally likely. If the cups were not randomized, the p-value does not mean what it says.

The code

# Fisher's lady-tasting-tea design: 8 cups, 4 with milk poured first.
# Rows: what was poured first (milk, tea). Columns: what the taster said.
from math import comb
from scipy.stats import fisher_exact

for correct in (4, 3):
    table = [[correct, 4 - correct],
             [4 - correct, correct]]
    _, p = fisher_exact(table, alternative="greater")
    ways = sum(comb(4, k) * comb(4, 4 - k) for k in range(correct, 5))
    print(f"{correct} of 4 milk-first cups named, chance of {correct} or more by luck: p = {p:.4f}"
          f"  (= {ways}/{comb(8, 4)} by counting)")

The output

4 of 4 milk-first cups named, chance of 4 or more by luck: p = 0.0143  (= 1/70 by counting)
3 of 4 milk-first cups named, chance of 3 or more by luck: p = 0.2429  (= 17/70 by counting)

Command: python3 -I tea.py with SciPy 1.16.3. The counting column checks SciPy against plain arithmetic: one of the 70 guesses gets all four right, and 16 more get exactly three right.

What the numbers mean

Limits

• A small p-value says the data are unlikely under pure guessing. It does not measure how good her palate is, and it is not the chance that she has the skill.
• Eight cups is a very small test. With 3 of 4 correct, the test cannot separate a skilled taster from a lucky one; more cups would.
• The 0.05 cut-off is a convention, not a law. Choose the threshold before looking at the data.
• The one-sided test fits because only "better than guessing" was the question. A two-sided test answers a different question and gives a different number.

The full method, including larger tables, two-sided p-values and when chi-square works instead, is in the Fisher's exact test paper.

On MCP Analytics you upload a CSV and tested statistical methods run on it. Each result shows the method used, and the same data gives the same numbers.

Sources

• SciPy documentation, scipy.stats.fisher_exact: the function used above, its alternative="greater" option, and its reference to Fisher, The Design of Experiments (1935).
• Wikipedia, Lady tasting tea: the eight-cup design and the 1-in-70 and 17-in-70 counts.