Did Your A/B Test
Actually Win?

Statistical significance in A/B testing indicates whether the difference between your control and variant is likely due to actual performance differences rather than random chance.

A test is typically considered statistically significant when the p-value is below 0.05 (95% confidence level), meaning there's less than a 5% probability that the observed difference occurred by chance.

MCP Analytics calculates this automatically using t-tests for continuous metrics (like revenue) and z-tests for proportions (like conversion rates), and provides plain-English interpretation so you don't need a statistics degree to understand the results.

You should stop an A/B test when you meet one of these criteria:

• Statistical significance reached: You've hit p < 0.05 AND collected your pre-determined minimum sample size
• Maximum duration reached: You've hit the maximum test duration you set beforehand (usually 2-4 weeks)
• Sequential analysis allows early stopping: Methods like Bayesian sequential testing indicate you can stop with high confidence

Warning: Never stop a test just because it looks like a winner early on. This practice, called "peeking," dramatically inflates false positive rates. If you check results 10 times during a test at p=0.05, your actual false positive rate is closer to 30%.

MCP Analytics provides guidance on when your test has sufficient data to make a reliable decision.

Sample size depends on four key factors:

• Baseline conversion rate: Your current metric (e.g., 5% conversion rate)
• Minimum detectable effect (MDE): The smallest improvement worth detecting (e.g., 10% relative lift)
• Statistical power: Typically 80% - the probability of detecting a real effect
• Significance level: Typically 95% (alpha = 0.05)

Example: To detect a 10% relative lift from a 5% baseline conversion rate with 80% power and 95% confidence, you need approximately 31,000 visitors per variant (62,000 total).

MCP Analytics includes a built-in sample size calculator that accounts for all these factors.

Frequentist A/B testing (traditional approach):

• Uses p-values and confidence intervals
• Answers: "Is there a statistically significant difference?"
• Requires fixed sample size determined in advance
• Can't be "peeked" at without inflating false positives

Bayesian A/B testing:

• Calculates the probability that one variant beats another
• Answers: "What's the probability B is better than A?" (e.g., 94%)
• Allows continuous monitoring without peeking problems
• Provides more intuitive results for business decisions

MCP Analytics provides both approaches so you can make the best decision for your situation. Bayesian is often preferred for its intuitive probability statements.

The standard threshold is p < 0.05, meaning you can be 95% confident the result isn't due to chance. However, the appropriate threshold depends on your situation:

• p < 0.10 (90% confidence): Acceptable for low-risk changes or exploratory tests
• p < 0.05 (95% confidence): Standard threshold for most business decisions
• p < 0.01 (99% confidence): Recommended for high-stakes decisions like major redesigns or pricing changes

Important: Statistical significance alone doesn't guarantee practical significance. A test might show p=0.01 but only a 0.1% improvement - technically significant but not worth implementing. Always consider effect size and business impact alongside p-values.

A 95% confidence interval shows the range where the true effect likely falls. For example, if your A/B test shows a lift of +15% with a 95% CI of [+8%, +22%], you can be confident the real improvement is between 8% and 22%.

Key interpretations:

• If the CI doesn't include zero (or 1.0 for ratios), the result is statistically significant
• The width indicates precision - narrower intervals mean more reliable estimates
• For business decisions, focus on the lower bound as a conservative estimate

Example: If CI = [+2%, +25%], you can be confident you'll see at least a 2% improvement. If CI = [-5%, +10%], the result is not significant because the interval includes zero (no effect).

Yes, but with caution. Running multiple tests on the same users creates interaction effects that can skew results. Best practices include:

• Mutually exclusive traffic: Allocate different user segments to different tests that might interact
• Multiple comparison corrections: Apply Bonferroni or FDR corrections when testing many variants
• Track experiment exposure: Know which users are in which experiments
• Consider multivariate testing: Use MVT for related changes to measure interactions

MCP Analytics supports multi-variant testing with up to 10 variants and automatically adjusts for multiple comparisons to maintain valid statistical conclusions.