Group comparison

A simulated example, with the code and its output

The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.

Question: Does an outcome differ between three groups, by how much, and which pairs of groups differ?

Inputs: Simulated data, not real customers. Seed 7 makes 40 values per group for groups A, B and C. Built in by construction: true means A=50, B=50, C=55, all with a spread (standard deviation) of 10, so A and B truly do not differ and C is truly 5 higher. The power checks use seeds 1000 to 1299 (300 seeds) with 10 or 40 values per group; in the 'no difference' runs all three means are set to 50.

Method: One-way ANOVA tests whether the group means are all equal, using the F statistic; the p-value is the chance of a gap this large if all three groups really had the same mean. The Kruskal-Wallis test is the nonparametric check beside it: it compares ranks instead of means, so it does not assume a bell-shaped spread. To see which pairs differ, each pair gets a Welch t-test (which does not assume equal spread), and the three p-values are adjusted with the Holm method so that running three tests does not inflate false alarms. Power is the share of seeds in which the ANOVA gives p below 0.05.

Result: In seed 7 the group means came out A 46.05, B 50.42 and C 54.14, against built-in 50, 50 and 55. ANOVA gave F=9.01 with p=0.0002 and Kruskal-Wallis gave p=0.0003, so both flagged that the groups are not all equal. By pair, A versus C differed by 8.09 (Holm p=0.0001). A versus B differed by 4.38 with a raw p of 0.0271 but a Holm p of 0.0541, and B versus C by 3.71 with p=0.0597. A and B were built equal, so the 4.38 gap is sampling luck, and it shows why unadjusted pair tests mislead. Across 300 seeds with 40 per group the ANOVA flagged the built-in difference in 0.61 of seeds, and with 10 per group in 0.18. With no difference built in it flagged 0.06 and 0.05, near the 0.05 expected.

Limit: The groups here are three equal-sized, bell-shaped samples with the same spread, with nothing else going on. It does not show unequal group sizes, outliers, repeated measures on the same people, or how the tool reports its results. Holm is one choice of adjustment; other adjustments can give different p-values.

Python script (numpy and scipy only):

import numpy as np
from scipy import stats
# Simulated data. Built in: true group means A=50, B=50, C=55; common spread (SD) 10.
rng = np.random.default_rng(7)
means = {"A": 50, "B": 50, "C": 55}
g = {k: rng.normal(m, 10, 40) for k, m in means.items()}
for k, v in g.items():
    print(k, "n=%d mean=%.2f sd=%.2f" % (len(v), v.mean(), v.std(ddof=1)))
F, p = stats.f_oneway(*g.values())
print("ANOVA: F=%.2f p=%.4f" % (F, p))
H, pk = stats.kruskal(*g.values())
print("Kruskal-Wallis: H=%.2f p=%.4f" % (H, pk))
# which pairs differ: Welch t-tests with Holm adjustment
pairs = [("A", "B"), ("A", "C"), ("B", "C")]
raw = [stats.ttest_ind(g[a], g[b], equal_var=False).pvalue for a, b in pairs]
order = np.argsort(raw)
adj = [0] * 3
run = 0
for rank, i in enumerate(order):
    run = max(run, min(1, raw[i] * (3 - rank)))
    adj[i] = run
for (a, b), r, q in zip(pairs, raw, adj):
    d = g[b].mean() - g[a].mean()
    print("%s vs %s: diff=%.2f raw p=%.4f Holm p=%.4f" % (a, b, d, r, q))
# repeat over 300 seeds: how often does ANOVA p<0.05 with n=40, and with n=10?
for n in (10, 40):
    for label, mm in (("C=55", means), ("no difference", {"A": 50, "B": 50, "C": 50})):
        hits = 0
        for s in range(300):
            r = np.random.default_rng(1000 + s)
            d = [r.normal(m, 10, n) for m in mm.values()]
            hits += stats.f_oneway(*d).pvalue < 0.05
        print("n=%d per group, %s: p<0.05 in %.2f of 300 seeds" % (n, label, hits / 300))

Printed output from running the code above:

A n=40 mean=46.05 sd=8.21
B n=40 mean=50.42 sd=9.13
C n=40 mean=54.14 sd=8.21
ANOVA: F=9.01 p=0.0002
Kruskal-Wallis: H=16.57 p=0.0003
A vs B: diff=4.38 raw p=0.0271 Holm p=0.0541
A vs C: diff=8.09 raw p=0.0000 Holm p=0.0001
B vs C: diff=3.71 raw p=0.0597 Holm p=0.0597
n=10 per group, C=55: p<0.05 in 0.18 of 300 seeds
n=10 per group, no difference: p<0.05 in 0.05 of 300 seeds
n=40 per group, C=55: p<0.05 in 0.61 of 300 seeds
n=40 per group, no difference: p<0.05 in 0.06 of 300 seeds

What's in the report?

Does the outcome differ between groups, by how much, and which pairs of groups differ, with t-tests or ANOVA and the nonparametric check beside them.