A/B Testing Power Analysis for Product Teams

1. Introduction

1.1 The Problem

A test with too few users usually returns "no significant difference", which is not evidence of no effect. A test that is stopped the moment it looks good often reports an effect that is not there. Both waste engineering time and lead to decisions that do not replicate.

1.2 What This Paper Covers

• The sample-size formula for comparing two proportions and how it scales
• Why repeated looks raise the false positive rate, with a simulation
• Sequential testing, Bayesian summaries and bandits, and what each is for
• A short planning checklist

2. Background

2.1 Error Rates and Power

The significance level alpha is the probability of declaring an effect when there is none; power is the probability of detecting an effect of a given size when it exists. Conventional choices are alpha = 0.05 and power of 0.8, but they are conventions, and the right values depend on the cost of each kind of mistake.

2.2 Minimum Detectable Effect

The minimum detectable effect (MDE) is the smallest true difference you want the test to find with the chosen power. It is a business choice: the smallest improvement worth acting on, not the improvement you hope for. A smaller MDE needs a larger sample.

3. Approach

3.1 Sample Size for Two Proportions

For a two-sided test comparing a baseline conversion rate p₁ with p₂ = p₁ plus the MDE, the users needed per variant are approximately

n = (z(1 - alpha/2) + z(power))^2 * (p1(1 - p1) + p2(1 - p2)) / (p2 - p1)^2

The denominator is the squared difference, so required n scales with the inverse square of the effect. The result is a planning figure: real data can be noisier than the formula assumes, users may not be independent (for example repeat visitors), and the tool you use may apply a different variance formula.

3.2 Peeking

Checking a running test and stopping when p drops below 0.05 gives each look a chance to produce a false positive. The looks share the same observations, so they are correlated, and the combined error rate depends on how many looks there are and when. It is below the figure you would get by treating looks as independent, and far above 5%.

3.3 Valid Ways to Monitor

• Fixed horizon: decide the sample size, look once at the end.
• Group sequential designs: plan a small number of looks and spend the error rate across them with an alpha-spending function (for example O'Brien-Fleming style boundaries), as in clinical trials.
• Always-valid methods: mixture sequential probability ratio tests give p-values and intervals that stay valid however often you look, and need no planned schedule (Johari and colleagues, 2017).

3.4 Bayesian Summaries and Bandits

A Bayesian analysis reports the probability that one variant is better given the data and a prior. That probability is a different quantity from a frequentist p-value; a posterior probability of 95% is not the same decision rule as significance at the 5% level, and stopping a Bayesian test whenever a threshold is crossed still has error properties that should be checked by simulation. Multi-armed bandits shift traffic toward better-looking variants to reduce the cost of testing; they optimize results during the run, not clean measurement of the difference, and are a different tool for a different goal.

4. Reproducible Calculations

4.1 Sample Size and the Inverse-Square Rule

import numpy as np
from scipy import stats

def n_per_variant(p1, relative_lift, alpha=0.05, power=0.80):
    """Users per variant for a two-sided two-proportion z-test (unpooled variance, as in the formula above)."""
    p2 = p1 * (1 + relative_lift)
    z = stats.norm.ppf(1 - alpha / 2) + stats.norm.ppf(power)
    return z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2

print("baseline  relative lift  absolute effect  users per variant")
for p1, lift in [(0.05, 0.20), (0.05, 0.10), (0.05, 0.05), (0.05, 0.02), (0.10, 0.10), (0.10, 0.05)]:
    print(f"{p1:8.0%}  {lift:13.0%}  {p1 * lift * 100:13.2f} pp  {np.ceil(n_per_variant(p1, lift)):17,.0f}")

baseline  relative lift  absolute effect  users per variant
      5%            20%           1.00 pp              8,155
      5%            10%           0.50 pp             31,231
      5%             5%           0.25 pp            122,121
      5%             2%           0.10 pp            752,700
     10%            10%           1.00 pp             14,749
     10%             5%           0.50 pp             57,760

Halving the relative lift from 10% to 5% at a 5% baseline moved the requirement from about 31,000 to about 122,000 per variant, close to fourfold, and a 2% lift needed about 753,000. For the same relative lift, a 10% baseline needed fewer users than a 5% baseline (about 14,700 against 31,200 for a 10% lift), because the absolute difference is twice as large.

4.2 What Peeking Does

The simulation runs 10,000 experiments in which there is no real effect at all, with 2,800 users per arm, and applies two rules at several numbers of equally spaced looks: stop at the first look with p below 0.05, or use a mixture sequential test and stop when its always-valid p-value falls below 0.05.

import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)
N_EXPERIMENTS, N_PER_ARM, TAU = 10_000, 2_800, 0.1   # users per arm, and the prior sd for the mixture test

# No real effect: both arms ~ Normal(0, 1).  We watch the per-pair difference d_i = treatment_i - control_i ~ Normal(0, 2).
d = rng.normal(0.0, np.sqrt(2.0), size=(N_EXPERIMENTS, N_PER_ARM))
n = np.arange(1, N_PER_ARM + 1)
running_mean = np.cumsum(d, axis=1) / n
z = running_mean * np.sqrt(n) / np.sqrt(2.0)          # ordinary z statistic at each sample size
p_fixed = 2 * stats.norm.sf(np.abs(z))

# Mixture sequential probability ratio test (normal data, known variance s2, N(0, TAU^2) mixture on the effect)
s2 = 2.0
log_lr = 0.5 * np.log(s2 / (s2 + n * TAU ** 2)) + (n ** 2 * TAU ** 2 * running_mean ** 2) / (2 * s2 * (s2 + n * TAU ** 2))
p_always_valid = np.minimum.accumulate(np.minimum(1.0, np.exp(-log_lr)), axis=1)

print("looks  schedule                          fixed-sample test, stop at first p < 0.05   mixture test (running minimum over all sample sizes)")
for looks in (1, 2, 4, 14, 28):
    idx = np.linspace(N_PER_ARM / looks, N_PER_ARM, looks).astype(int) - 1
    fixed = (p_fixed[:, idx] < 0.05).any(axis=1).mean()
    mixture = (p_always_valid[:, idx] < 0.05).any(axis=1).mean()
    print(f"{looks:5d}  {'equally spaced looks':32s}  {fixed:41.3f}   {mixture:24.3f}")

looks  schedule                          fixed-sample test, stop at first p < 0.05   mixture test (running minimum over all sample sizes)
    1  equally spaced looks                                                  0.051                      0.024
    2  equally spaced looks                                                  0.084                      0.024
    4  equally spaced looks                                                  0.129                      0.024
   14  equally spaced looks                                                  0.222                      0.024
   28  equally spaced looks                                                  0.280                      0.024

With one look the fixed-sample test errs at the nominal 5%. With repeated looks the false positive rate grew steadily, to 28% at 28 looks. The mixture test's always-valid p-value is the running minimum over every sample size, so it gives the same 2.4% whatever the number of looks, and 2.4% is below 5% because the method is conservative. The price is lower power for a given sample, a general property of such methods that this run does not measure, and its tuning parameter (here the prior standard deviation of 0.1) affects that trade.

5. Practical Guidance

6. Conclusion

Sample size is a plan, not an afterthought: it follows from the effect worth detecting and grows with the inverse square of that effect. Looking early and stopping on a good result is the most common way to break an experiment's error rate; the simulation shows the size of the damage and one method that avoids it. Decide the plan first, then choose the monitoring method that fits it.