Causal Inference Methods: Complete Guide to Measuring What Actually Works

Correlation is not causation. Everyone knows this. Very few people know what to do about it. When your marketing campaign runs at the same time revenue spikes, did the campaign cause the spike, or was it seasonal demand? When customers who use your premium feature churn less, is the feature reducing churn, or do loyal customers just happen to use more features?

Causal inference methods answer these questions rigorously. They are the difference between "these two things moved together" and "this thing caused that thing." This guide covers methods from the gold standard (randomized A/B tests) to quasi-experimental methods for situations where you cannot randomize. Each section tells you when a method works, when it fails, and links to a full deep-dive article.

Find the method for your question

Start with the question you are asking. Each guide says when its method does not fit.

A Worked Example: Difference-in-Differences on Made-Up Data

This example shows one method of the many listed below. It is not representative of the other methods, and the data are made up. For the method itself, see the difference-in-differences guide.

Question: Did a new feature raise weekly orders per account for the accounts that got it, compared with accounts that did not?

Inputs: Simulated weekly orders for 80 accounts in each group, before and after the change. The groups start at different levels on purpose (about 20 and 18 orders). Both groups are set to rise by 1.5, and the group with the feature gets 3.0 more. Noise is added in both periods. Seed 31415.

Method: Difference-in-differences compares two changes: the change in the group that got the feature and the change in the group that did not. It assumes the groups would have had alike trends if there were no feature. The standard error shows how much the estimate would be expected to move with a different sample.

import numpy as np

SEED = 31415
rng = np.random.default_rng(SEED)
n = 80
common_time_change = 1.5
simulated_effect = 3.0
t_base = rng.normal(20.0, 4.0, n)
c_base = rng.normal(18.0, 4.0, n)
t_before = t_base + rng.normal(0.0, 2.0, n)
c_before = c_base + rng.normal(0.0, 2.0, n)
t_after = (t_base + common_time_change + simulated_effect
           + rng.normal(0.0, 2.0, n))
c_after = (c_base + common_time_change
           + rng.normal(0.0, 2.0, n))

t_change = t_after - t_before
c_change = c_after - c_before
did = t_change.mean() - c_change.mean()
se = np.sqrt(t_change.var(ddof=1) / n + c_change.var(ddof=1) / n)

print(f"Simulated data, seed {SEED}; weekly orders per account, "
      f"{n} accounts in each group")
print("Built in: both groups rise by 1.5; "
      "the group with the feature gets 3.0 more")
print(f"Feature group: before {t_before.mean():.3f}, "
      f"after {t_after.mean():.3f}, change {t_change.mean():.3f}")
print(f"Other group: before {c_before.mean():.3f}, "
      f"after {c_after.mean():.3f}, change {c_change.mean():.3f}")
print(f"Difference-in-differences estimate: {did:.3f} orders per "
      f"account (standard error {se:.3f})")
print(f"Rough 95% range: {did - 1.96 * se:.3f} to "
      f"{did + 1.96 * se:.3f}")

Simulated data, seed 31415; weekly orders per account, 80 accounts in each group
Built in: both groups rise by 1.5; the group with the feature gets 3.0 more
Feature group: before 20.228, after 24.146, change 3.918
Other group: before 17.802, after 19.429, change 1.627
Difference-in-differences estimate: 2.292 orders per account (standard error 0.505)
Rough 95% range: 1.301 to 3.282

Result: The group with the feature rose by 3.92 orders per account and the other group by 1.63, so the difference-in-differences estimate is 2.29 (range 1.30 to 3.28). The data were built with an effect of 3.0 and the same trend in both groups, so this shows the method on data where its main assumption (alike trends) holds by construction. The range includes 3.0, but the estimate is below it, which is what noise of this size does.

Limit: Code made the data. The example cannot show whether real groups would have had alike trends, and it does not show the effect of a real feature.

Quick Comparison

Method Deep-Dives

Decision Guide: Which Method Do You Need?

How These Methods Relate

Causal inference methods differ in how much they rely on assumptions:

1. Randomized experiments (A/B testing, Bayesian A/B) -- the gold standard. Randomization eliminates confounders by design.
2. Quasi-experimental methods (DiD, RD, synthetic control, causal impact) -- exploit natural variation or policy changes to approximate randomization. Strong when assumptions hold.
3. Matching and adjustment methods (propensity scores, ANCOVA, IV) -- use statistical techniques to adjust for confounders. Quality depends entirely on having the right covariates or instruments.

In practice, most businesses start with A/B testing for digital experiments and use DiD or Causal Impact for interventions that cannot be randomized (pricing changes, market launches, policy shifts). Propensity score matching fills the gap when you have observational data and rich customer profiles. The other methods are specialized tools for specific situations.

Causal Inference for Business Decisions

All Causal Inference Guides