t-Test Assumptions: What Breaks Them & Fixes

A common pattern in A/B testing: a team runs the test, gets a p-value, and checks the assumptions afterwards, if at all.

Here's the problem: assumption violations don't announce themselves. Your test runs. You get a p-value. It looks significant. You ship the change. But if your data violated homoscedasticity or normality, the false positive rate can be above the rate you asked for. The worked example below shows one setup where it is. You may have made a business decision based on noise.

Pricing changes, feature releases, and marketing spend are often decided on tests like these. The fix is to check assumptions before you run the test, know which violations matter, and use the right variant when assumptions fail.

Worked Example (Simulated): Same Mean, Unequal Spread, Unequal Group Sizes

Question: When the two groups have the same mean, how often does each test find a difference?

Inputs: Simulated data, seed 20261029. The smaller group has 30 values and a standard deviation (SD) of 2.0. The larger group has 50 values and an SD of 1.0. Both population means are 0, so any difference the tests find is a false positive. The run makes 20,000 replications. The result depends on this pairing: the smaller group is the one with more spread.

Method: In each replication, the script draws independent normal values for both groups. It runs two-sided Student and Welch t-tests at alpha 0.05 and counts the share of replications with p below alpha.

"""Illustrate Type I error for unequal variances and unequal group sizes."""
import numpy as np
from scipy.stats import t

seed = 20261029
reps = 20000
n_small = 30
n_large = 50
sd_small = 2.0
sd_large = 1.0
alpha = 0.05
rng = np.random.default_rng(seed)

# The null is true: both populations have mean zero. The smaller group
# has the larger standard deviation. Rows are independent replications.
x = rng.normal(0.0, sd_small, size=(reps, n_small))
y = rng.normal(0.0, sd_large, size=(reps, n_large))
mx = x.mean(axis=1)
my = y.mean(axis=1)
vx = x.var(axis=1, ddof=1)
vy = y.var(axis=1, ddof=1)

# Student's pooled-variance two-sided t test.
df_student = n_small + n_large - 2
pooled_var = ((n_small - 1) * vx + (n_large - 1) * vy) / df_student
se_student = np.sqrt(pooled_var * (1 / n_small + 1 / n_large))
t_student = (mx - my) / se_student
p_student = 2 * t.sf(np.abs(t_student), df_student)

# Welch's two-sided t test with the Satterthwaite degrees of freedom.
a = vx / n_small
b = vy / n_large
se_welch = np.sqrt(a + b)
df_welch = (a + b) ** 2 / (a ** 2 / (n_small - 1) + b ** 2 / (n_large - 1))
t_welch = (mx - my) / se_welch
p_welch = 2 * t.sf(np.abs(t_welch), df_welch)

print(f"Simulated data, seed {seed}")
print(f"Replications: {reps}; group sizes: {n_small} and {n_large}")
print(f"Population means: 0 and 0; "
      f"standard deviations: {sd_small} and {sd_large}")
print(f"Two-sided tests; alpha: {alpha}; smaller group has larger spread")
print(f"Student rejection rate: {np.mean(p_student < alpha):.4f}")
print(f"Welch rejection rate: {np.mean(p_welch < alpha):.4f}")

Printed output:

Simulated data, seed 20261029
Replications: 20000; group sizes: 30 and 50
Population means: 0 and 0; standard deviations: 2.0 and 1.0
Two-sided tests; alpha: 0.05; smaller group has larger spread
Student rejection rate: 0.0938
Welch rejection rate: 0.0491

Result: Student's test rejects in 0.0938 of replications. Welch's test rejects in 0.0491 of replications. With equal means, the share of rejections should be near alpha, 0.05. Student's is above it in this setup and Welch's is close to it.

Limit: This is one simulated setup. It does not show what will happen with other data or group sizes. It does not show the case where the larger group has the larger SD.

Sources

Cited for who published each test and when, not for any number on this page.

• Student (W. S. Gosset) (1908). The probable error of a mean. Biometrika 6. doi 10.2307/2331554
• Welch B L (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika 34. doi 10.1093/biomet/34.1-2.28

The 3 Critical Assumptions (And Why They Exist)

The t-test was published by William Sealy Gosset, writing as Student, in 1908 (Student, 1908). It works under three conditions:

To test whether your own outcome differs between groups, and which groups differ, try the free group comparison tool.

1. Independence: Each observation is independent of others
2. Normality: Data in each group follows a normal distribution
3. Homoscedasticity: Equal variance across groups

Before we draw conclusions about which variant won, let's understand why these assumptions exist and what happens when they break.

Independence: The Non-Negotiable Assumption

This one you can't violate. If observations aren't independent, if one user's behavior affects another's, or if you measured the same users multiple times without accounting for clustering. The t-test formula is mathematically wrong. Period.

Random assignment can help reduce confounding. Also check for clusters, repeat measures, and cases where one user's outcome can affect another's. If you let users self-select into groups, or assigned based on geography where network effects exist, you violated independence.

Quick check: Can you draw a line from observation A to observation B showing how A might influence B? If yes, you have dependent data. Use mixed-effects models or clustered standard errors instead.

Normality: The Assumption That Bends

Normality matters less than you think, with caveats. The Central Limit Theorem says that sampling distributions of means approach normality even when raw data doesn't. But "approach" depends on sample size and the degree of non-normality.

With mild skewness and a reasonable sample size per group, the t-test is often robust. With severe skewness, heavy tails, or outliers, you need much larger samples or a different test entirely.

Homoscedasticity: The Assumption Everyone Ignores

Equal variance across groups matters. When variances differ and sample sizes are unbalanced, Student's t-test becomes liberal (inflates Type I error) or conservative (loses power) depending on which group has larger variance.

One fix is to use Welch's t-test (Welch, 1947). It doesn't assume equal variances and applies the Satterthwaite correction for degrees of freedom.

Testing Normality: What Actually Works

You need two tools: visual assessment (Q-Q plots) and formal tests (Shapiro-Wilk). Neither alone gives the full picture.

Q-Q Plots: Your First Line of Defense

A Q-Q plot (quantile-quantile plot) plots your data quantiles against theoretical normal distribution quantiles. If data is normal, points fall on a straight line.

What to look for:

• S-curve pattern: Heavy tails (more extreme values than normal distribution predicts)
• Inverted S-curve: Light tails (fewer extreme values)
• Points curve up at right: Right skew (long tail toward high values)
• Points curve down at left: Left skew (long tail toward low values)
• Outliers: Points far from the line at either end

Visual assessment shows you the type and severity of a violation. A formal test just says "rejected" without context.

Shapiro-Wilk Test: Formal Hypothesis Testing

The Shapiro-Wilk test is one formal way to check for a departure from normality.

# Python example
from scipy.stats import shapiro

# Test normality for control group
stat, p_value = shapiro(control_data)
print(f"Shapiro-Wilk statistic: {stat:.4f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Reject normality assumption")
else:
    print("Fail to reject normality")

Interpretation: If p < 0.05, reject normality. But here's the trap: with large samples, Shapiro-Wilk can become very sensitive. It can reject normality for small departures that matter little for the t-test.

That's why you need both visual and formal assessment. Q-Q plots show severity. Shapiro-Wilk provides a statistical threshold. Together, they give you the full picture.

The n > 30 Myth Debunked

Does the n > 30 rule really make normality violations okay? No. This rule is an oversimplification.

What's actually true: The Central Limit Theorem says the sampling distribution of the mean approaches normality as n increases. The rate of approach depends on the underlying distribution's shape.

How the sample size you need changes with the distribution:

• Symmetric, light-tailed: smaller samples are often enough
• Moderate skewness: larger samples are needed
• Severe skewness: much larger samples are needed
• Heavy tails or outliers: much larger samples, or use robust methods
• Extreme skewness (e.g., income data): the CLT may not help at any reasonable n

The fix: check your actual data. Don't rely on sample size alone.

Homoscedasticity: When Unequal Variance Matters

Equal variance isn't just a theoretical nicety. When it's violated, Student's t-test produces incorrect p-values and confidence intervals. The severity depends on two factors: the variance ratio and whether sample sizes are balanced.

Levene's Test for Equal Variance

Levene's test compares group variances. A small p-value is evidence against equal variances. A larger p-value does not show that the variances are equal. Use Welch's t-test by default.

# Python example
from scipy.stats import levene

# Test equal variance assumption
stat, p_value = levene(control_data, treatment_data)
print(f"Levene statistic: {stat:.4f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Variances are unequal - use Welch's t-test")
else:
    print("No evidence against equal variances")

If Levene's p-value is below 0.05, or the variances look clearly different, use Welch's t-test.

Visual Check: Side-by-Side Boxplots

Before running formal tests, create boxplots for each group. If one box is noticeably wider than the other, the variances differ. Look at the interquartile range (IQR), the height of the box, as a proxy for variance.

Calculate the variance ratio manually. The further it is from 1, the more the variances differ:

import numpy as np

var_control = np.var(control_data, ddof=1)
var_treatment = np.var(treatment_data, ddof=1)
variance_ratio = var_treatment / var_control

print(f"Variance ratio: {variance_ratio:.2f}")

What Unequal Variance Actually Does

Here's the damage pattern:

• Balanced samples + unequal variance: Minimal impact on Type I error, slight power loss
• Unbalanced samples, smaller group has larger variance: Liberal test (inflated Type I error; see the worked example above)
• Unbalanced samples, larger group has larger variance: Conservative test (Type I error below the nominal rate, reduced power)

This is why sample size balance matters. With equal n, the t-test is less sensitive to variance differences. With unequal n, variance differences can cause problems.

Student's vs Welch's t-Test: When the Satterthwaite Correction Saves You

The difference between Student's (1908) and Welch's (1947) t-test comes down to degrees of freedom and how they pool variance.

Student's t-Test: The Pooled Variance Approach

Student's t-test assumes equal variances and pools them:

s_pooled² = ((n₁-1)s₁² + (n₂-1)s₂²) / (n₁ + n₂ - 2)
df = n₁ + n₂ - 2

This works when variances are equal. When they're not, the pooled estimate is wrong and df is incorrect.

Welch's t-Test: The Satterthwaite Correction

Welch's t-test doesn't pool variances. Instead, it uses the Satterthwaite approximation for degrees of freedom:

df_welch = (s₁²/n₁ + s₂²/n₂)² / ((s₁²/n₁)²/(n₁-1) + (s₂²/n₂)²/(n₂-1))

This formula adjusts df based on the actual variance structure. When variances differ, df_welch is typically lower than df_student, making the critical value larger and the test more conservative (protective).

Practical Comparison: When It Matters

The worked example above shows one illustrative simulation with normal data and equal group means. It gives the code, seed, group sizes, spread, test settings and results. Student's test rejects in 0.0938 of replications and Welch's test in 0.0491. These are results for this setup.

Decision Rule: Which Test to Use

Use Welch's t-test when:

• Levene's test p < 0.05 (evidence of different variances)
• The variances look clearly different
• Sample sizes differ
• You're unsure whether the variances are equal

Use Student's t-test when:

• You have strong evidence of equal variances
• Sample sizes are balanced (n₁ = n₂) and the variances are similar
• You specifically need the higher power from pooled variance estimates

Welch's test does not assume equal variances; Student's test does.

When Violations Matter More, and When They Matter Less

Severity of the violation, sample size, and group balance decide how much a violation matters. The worked example above covers one case: unequal variances with unequal group sizes. These general points are not tied to a run on this page:

• Mild skew with equal group sizes: the t-test is often little affected.
• Severe skew: the t-test can lose power or err. Match the test to your question. Mann-Whitney answers a different question (whether one group tends to produce larger values). If your question is about means and the data are heavily skewed, a bootstrap or permutation test of the difference in means keeps the question and drops the normality assumption.
• Outliers: they can cost power, so investigate them before trusting a non-significant result.

Non-Parametric Alternative: When to Use Mann-Whitney U

When normality is hopeless and transformations don't work, the Mann-Whitney U test (also called Wilcoxon rank-sum test) is your non-parametric alternative.

How Mann-Whitney Works

Instead of comparing means, Mann-Whitney compares ranks. It tests whether observations from one group tend to be larger than observations from the other group.

# Python example
from scipy.stats import mannwhitneyu

# Non-parametric test (doesn't assume normality)
stat, p_value = mannwhitneyu(control_data, treatment_data, alternative='two-sided')
print(f"Mann-Whitney U statistic: {stat:.2f}")
print(f"p-value: {p_value:.4f}")

What it tests: The null hypothesis is that a randomly selected value from group 1 is equally likely to be greater than or less than a randomly selected value from group 2.

When to Use Mann-Whitney

• Severe non-normality: Heavy skew, multimodal distributions, or distributions with no clear shape
• Ordinal data: Likert scales, satisfaction ratings, rankings
• Outliers present: When you can't remove them and they're legitimate data
• Small samples with clearly non-normal data: when a transformation does not help

The Power Trade-off

Here's the cost: Mann-Whitney can have less power than the t-test when the data are actually normal, so you may need a somewhat larger sample to detect the same effect.

Whether that cost is worth paying depends on how clear the violation is and on the question you are asking.

Common Misconception: Mann-Whitney Tests Medians

This is wrong. Mann-Whitney does not test whether medians differ (unless distributions have identical shapes). It tests stochastic dominance, whether values from one group tend to be systematically larger.

If you specifically want to test median differences, use a bootstrap test or quantile regression instead.

Sample Size Matters: When Small n Makes Violations Matter More

What's your sample size? That's the question that determines how seriously to take assumption violations.

The Small Sample Problem

With small samples, two problems compound:

1. Low power to detect assumption violations: Shapiro-Wilk and Levene's tests have little power with small samples, so they can fail to flag real violations
2. Greater impact from violations: The Central Limit Theorem hasn't kicked in yet, so departures from normality directly affect the sampling distribution

This creates a trap: tests can't reliably detect violations that would matter most.

Solution for small n:

• Rely heavily on visual assessment (Q-Q plots, boxplots)
• Default to Welch's t-test unless you have prior evidence of equal variances
• Consider non-parametric tests if data looks clearly non-normal
• Better yet: increase your sample size before running the test

The Large Sample Regime

With large samples, the Central Limit Theorem provides protection against normality violations. But here's the paradox: formal tests become hypersensitive.

Shapiro-Wilk can reject normality for small departures that matter little for the t-test. Levene's test can flag variance differences that don't meaningfully affect Type I error.

Solution for large n:

• Trust visual assessment over formal tests
• Focus on practical significance: how different are the variances? Does the Q-Q plot show severe departures?
• Welch's t-test does not assume equal variances, so it is a reasonable default

Sample Size and Effect Size Interaction

Here's something practitioners miss: assumption violations matter more when effects are small.

With a large effect (Cohen's d = 1.0), even a sloppy test will detect it. With a small effect (d = 0.2), violations can obscure real differences or create false ones.

Calculate your minimum detectable effect before running the test. If you need to detect small effects, you need cleaner data and more careful assumption checking.

from statsmodels.stats.power import TTestIndPower

# Calculate required sample size
effect_size = 0.5  # Cohen's d
alpha = 0.05
power = 0.80

n_required = TTestIndPower().solve_power(
    effect_size=effect_size, alpha=alpha, power=power
)
print(f"Required n per group: {n_required:.0f}")

Consider a Different Metric

For a skewed metric such as revenue per session, the metric itself may be the issue. Consider splitting it:

• Conversion rate: the share of sessions that purchase (test with two-proportion z-test)
• Average order value: Mean revenue among converters only (subset data, then test)

This decomposition often reveals what's actually happening. Maybe conversion rates didn't change, but order values did (or vice versa).

Decision Flowchart: Which Test Should You Use?

Here's the practical decision tree for choosing the right test:

1. Check independence: Are observations independent?
   • No → Use paired t-test, mixed-effects model, or clustered SE
   • Yes → Continue
2. Check sample size: is each group large enough for your data?
   • No → Use exact tests or increase sample size
   • Yes → Continue
3. Check normality: Q-Q plot + Shapiro-Wilk
   • Severe violations → Try transformation
   • Mild violations → Continue
4. Check variance: Levene's test + variance ratio
   • Unequal (small Levene p-value or clearly different variances) → Use Welch's t-test
   • Equal → Use Student's t-test (or Welch's as safe default)
5. If transformation didn't fix normality:
   • Consider Mann-Whitney U (tests whether one group tends to produce larger values) or a permutation test of the mean difference
   • Or use bootstrap/permutation test

Welch's t-test is a reasonable default when your question is about the difference in means. When the data are heavily skewed or contaminated by outliers and your question is whether one group tends to produce larger values, Mann-Whitney is a closer match; for a question about means with skewed data, a bootstrap or permutation test of the difference in means keeps the question.

Quick Wins: Five Habits That Help

You don't need perfect data. You need to avoid the big mistakes. Here are five habits that help:

1. Consider Welch's t-Test

Unless you have strong prior evidence of equal variances, consider Welch's. It does not assume equal variances; Student's test does.

In R, Welch's is the default for t.test(). In Python, set equal_var=False.

2. Create Q-Q Plots

Visual assessment shows the type and severity of non-normality, which a formal test does not.

import scipy.stats as stats
import matplotlib.pyplot as plt

# Q-Q plot
stats.probplot(data, dist="norm", plot=plt)
plt.title("Q-Q Plot")
plt.show()

3. Balance Your Sample Sizes

Equal n per group makes the t-test less sensitive to unequal variances. Aim for n₁ = n₂ in your experimental design.

This is why proper A/B test implementation matters. Random assignment with equal allocation aims for balanced groups. Ad-hoc splits can cause trouble.

4. Calculate Sample Size Before Running the Test

Underpowered tests can produce unreliable results. Run a power analysis first, then collect enough data.

Plan for larger samples when effects are small or when assumptions are questionable.

5. Pre-Specify Your Analysis Plan

Decide which tests you'll use before looking at results. This prevents p-hacking (trying multiple tests until one gives p < 0.05).

Write down: "We'll use Welch's t-test. If Shapiro-Wilk p < 0.01 and skewness > 2, we'll log-transform first. If transformation doesn't work, we'll use Mann-Whitney (if our question is which group tends to be larger) or a permutation test of the mean difference."

Then stick to the plan.