Causal impact

The method, in full

The practical guide behind this tool: when it applies, how to read the output, and the traps that make it say the wrong thing.

Read the guide →

A simulated example, with the code and its output

The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.

Question: A page was relaunched at week 52. Did weekly signups move afterwards, by how much, and how does that compare with just subtracting the before average from the after average?

Inputs: Simulated data, not real signups. Seed 7 makes 78 weekly values: 52 before the change and 26 after. Built in by construction: a baseline of 200 signups, a drift of +0.5 per week that was already under way, then at week 52 a jump of +30 and an extra +0.4 per week of trend, so the true average weekly effect over the 26 weeks is 35.0. Noise is independent with a standard deviation of 12. The checks use seeds 1000 to 1299 (effect built in) and 5000 to 5299 (no effect built in), 300 seeds each.

Method: Interrupted time series, the method the page describes. One regression line is fitted to all 78 weeks with four terms: a starting level, a weekly trend, a jump at week 52, and a change in the weekly trend after week 52. The jump and the trend change come with 95% intervals from the regression. The effect against the expected path is the observed series minus the path the pre-change trend would have followed; its average over the 26 post weeks gets a 95% interval too. An interval that excludes zero is the sign that the change is bigger than week-to-week noise would give. The plain before and after difference in averages is shown for contrast.

Result: In seed 7 the jump at the change was 33.76 (95% interval 23.79 to 43.73) against 30.0 built in. The change in trend was 0.09 per week (interval -0.49 to 0.68), against 0.40 built in, so with 26 post weeks the trend change is not pinned down even though the jump is. The average weekly effect against the expected path was 34.94 (interval 25.83 to 44.05) against 35.0 built in. The plain after-minus-before difference in averages was 57.8, far too high, because the series was already drifting upward before the change. Across 300 seeds the interval held the built-in average effect in 0.95 of runs, excluded zero in 1.00 of runs with the effect built in, and excluded zero in 0.03 of runs with no effect built in.

Limit: The noise here is independent from week to week and the pre-change trend is a straight line that carries on unchanged. Real series have seasons, shocks and neighbours that change at the same time, and any of these can make the expected path wrong. The example does not show those cases, other outside events that coincide with the change, or how the tool reports its results. A result like this describes what moved after the change; it does not on its own prove the change caused it.

Python script (numpy and scipy only):

import numpy as np
from scipy import stats
# Simulated weekly signups. Built in: baseline 200, drift +0.5/week, then at week 52
# a jump of +30 and an extra +0.4/week trend change. Noise sd 12 (independent).
T0, N, SD = 52, 78, 12.0
JUMP, SLOPE_CH, BASE, DRIFT = 30.0, 0.4, 200.0, 0.5

def make(rng, jump=JUMP, sch=SLOPE_CH):
    t = np.arange(N); post = (t >= T0).astype(float)
    y = BASE + DRIFT * t + post * (jump + sch * (t - T0)) + rng.normal(0, SD, N)
    return t, post, y

def fit(t, post, y):
    X = np.c_[np.ones(N), t, post, post * (t - T0)]
    b, *_ = np.linalg.lstsq(X, y, rcond=None)
    res = y - X @ b; df = N - 4; s2 = res @ res / df
    cov = s2 * np.linalg.inv(X.T @ X); se = np.sqrt(np.diag(cov))
    q = stats.t.ppf(0.975, df)
    # average effect over the 26 post weeks = observed minus the path the pre-period trend predicts
    P = np.c_[np.ones(N - T0), t[T0:], np.zeros(N - T0), np.zeros(N - T0)]
    D = X[T0:] - P                      # difference in design rows = jump + trend change terms
    w = D.mean(0); avg = w @ b; avg_se = np.sqrt(w @ cov @ w)
    return b, se, q, avg, avg_se

rng = np.random.default_rng(7)
t, post, y = make(rng)
b, se, q, avg, avg_se = fit(t, post, y)
true_avg = JUMP + SLOPE_CH * (np.arange(N - T0)).mean()
print("Seed 7: 52 weeks before, 26 weeks after the change")
print("Built in: jump +30.0, trend change +0.40 per week, average weekly effect %.1f" % true_avg)
for name, i in (("Jump at the change", 2), ("Change in trend per week", 3)):
    print("%-26s %6.2f  95%% interval %6.2f to %6.2f" % (name, b[i], b[i] - q * se[i], b[i] + q * se[i]))
print("%-26s %6.2f  95%% interval %6.2f to %6.2f" % ("Average weekly effect", avg, avg - q * avg_se, avg + q * avg_se))
naive = y[T0:].mean() - y[:T0].mean()
print("Plain after-minus-before difference in means: %.1f (ignores the drift already under way)" % naive)

cover = hit = 0; fp = 0; R = 300
for s in range(R):
    _, _, ys = make(np.random.default_rng(1000 + s)); bb, ss, qq, a, ase = fit(t, post, ys)
    cover += (a - qq * ase <= true_avg <= a + qq * ase)
    hit += not (a - qq * ase <= 0 <= a + qq * ase)
    _, _, y0 = make(np.random.default_rng(5000 + s), 0.0, 0.0); bb, ss, qq, a, ase = fit(t, post, y0)
    fp += not (a - qq * ase <= 0 <= a + qq * ase)
print("Over %d seeds: interval held the built-in average effect in %.2f of runs" % (R, cover / R))
print("Interval excluded zero in %.2f of runs with the effect built in, %.2f of runs with no effect" % (hit / R, fp / R))

Printed output from running the code above:

Seed 7: 52 weeks before, 26 weeks after the change
Built in: jump +30.0, trend change +0.40 per week, average weekly effect 35.0
Jump at the change          33.76  95% interval  23.79 to  43.73
Change in trend per week     0.09  95% interval  -0.49 to   0.68
Average weekly effect       34.94  95% interval  25.83 to  44.05
Plain after-minus-before difference in means: 57.8 (ignores the drift already under way)
Over 300 seeds: interval held the built-in average effect in 0.95 of runs
Interval excluded zero in 1.00 of runs with the effect built in, 0.03 of runs with no effect

What's in the report?

Did the series change after a launch, price change or policy? The jump at the change, the change in trend after it, and the effect against the path the earlier trend projects.