ds@mcpanalytics.aiA data scientist you can email your data and question to, and get a reliable response. Not AI slop.A data scientist you can email. Not AI slop.
Worked example · Multiple comparisons

Which multiple-comparison correction do you actually need?

The reproducible worked example behind our multiple-comparisons lesson: 66 pairwise correlations on 1,599 wines, corrected four ways, cross-checked against R's own p.adjust.

The data

UCI Wine Quality (red), 1,599 wines, 12 chemistry variables, all 66 pairwise Pearson correlations. Real data, fetched from the UCI archive. No simulation.

The headline numbers

Swapping Bonferroni for Holm changed nothing here: both keep 43. The choice that moved the answer was family-wise error rate against false discovery rate, which moved 11 findings of 66.

Download and re-run it

Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.

What the review caught, and it was ours

The first draft of this document claimed the smallest p-values underflowed to zero in double precision. That was false: the smallest is 4.06e-220, and nothing underflowed. The claim had been imported from a note describing a different, earlier computation and repeated without testing it. It is corrected in the document, which now derives the real numbers live.

The honest limit

The 66 tests are not independent, because wine chemistry variables correlate. Bonferroni and Holm are valid under arbitrary dependence. Benjamini-Hochberg as used assumes independence or positive regression dependence, and the document says so.

The brief for an AI agent

If an AI assistant is choosing a method for you, it needs different things than you do: the discriminating condition, the questions this gets mistaken for, the failure modes that return a plausible number instead of an error, and figures it can cite rather than generate. We publish that as its own document, written to one bar: an agent asked whether to run this analysis should be able to answer from it alone, including saying no.

Read the agent brief

Where this came from

Watch the lesson Run it on your own data The brief for an AI agent All worked examples

Your turn

Bring your own data and the question you actually need answered.

CympleData Scientist Send me your data and question, I’ll send you the analytics. ds@mcpanalytics.ai