Causal Impact Explained (with Examples)

When you launch a marketing campaign, change pricing, or roll out a new feature, how do you know if it actually worked? Traditional analytics show what changed; causal impact analysis estimates what would have happened without the change, so the difference can be read as the effect. However, common mistakes can lead to misleading conclusions. This guide reviews the right and wrong approaches to causal impact analysis, helping you avoid common pitfalls and read a time series estimate of an event's effect with care.

In practice this is the method behind "did it work?" questions: measuring whether a campaign, price change, or feature launch actually moved the needle. See how it fits an experimentation workflow on our A/B testing page, or the broader marketing analytics use cases.

What is Causal Impact Analysis?

Causal impact estimates what might have happened without an event by using a time series model. The estimated effect depends on the model and its assumptions, including that the control was not changed by the event and that the model still fits after the event.

Unlike simple before-and-after comparisons, causal impact analysis builds a model of your data's behavior during a pre-intervention period, then projects what would have happened during the intervention period if nothing had changed. The difference between this projection and actual observations is the estimated effect, which holds only if the model's assumptions hold.

Many interventions cannot be tested with randomized experiments. Random assignment may be impractical when a change affects a whole region or system. If a group or region can be held out, compare it with the group that gets the change; if users can be assigned at random, an A/B test can give stronger evidence. Causal impact is one alternative when an A/B test is impractical.

Worked Example (Simulated): Did the Event Change the Outcome?

Question. Did an event change the outcome, and by how much per day, relative to a model fit before the event?

Inputs. Simulated data, seed 24. 60 pre days and 14 post days of an outcome and a control series. The control was built so the event does not touch it. The built-in effect is +4.0 outcome units per day, added to the outcome on the 14 post days.

Method. Fit a line on the pre days only, using an intercept, time and the control, then use it to predict the post days. The difference between actual and predicted is the estimated effect. This is a simple regression, not the Bayesian method this guide describes. It assumes the event did not change the control and that the link between the control and outcome stayed much the same.

import numpy as np

# Simulated data, seed 24
rng = np.random.default_rng(24)
n_pre, n_post = 60, 14
effect = 4.0                                 # built-in lift per post day
t = np.arange(n_pre + n_post)
control = 100 + 0.35 * t + 5 * np.sin(2 * np.pi * t / 7) + rng.normal(0, 2, len(t))
outcome = 30 + 0.6 * control + 0.12 * t + rng.normal(0, 2, len(t))
outcome[n_pre:] += effect

# Fit a linear counterfactual model on the pre-period only.
X = np.column_stack([np.ones(n_pre), t[:n_pre], control[:n_pre]])
beta, *_ = np.linalg.lstsq(X, outcome[:n_pre], rcond=None)
pre_fit = X @ beta
X_post = np.column_stack([np.ones(n_post), t[n_pre:], control[n_pre:]])
post_counterfactual = X_post @ beta
post_effect = outcome[n_pre:] - post_counterfactual
rmse = np.sqrt(np.mean((outcome[:n_pre] - pre_fit) ** 2))

print(f"Simulated data, seed 24; built-in effect: +{effect} outcome units per day")
print("Question: mean change in the outcome over the 14 post days,")
print("  relative to a model fit before the change?")
print("Inputs: 60 pre days and 14 post days; outcome and a simulated control.")
print("Method: least squares of outcome on intercept, time, control (pre days);")
print("  predict the post days.")
print(f"Pre-period RMSE: {rmse:.2f}")
mean_eff = np.mean(post_effect)
print(f"Mean estimated post effect: {mean_eff:.2f} outcome units per day")
print(f"Sum of estimated post effects: {np.sum(post_effect):.2f} outcome units")
print(f"Built-in effect minus estimate: {effect - mean_eff:.2f}")
print("Limit: a simple simulated regression, not the Bayesian structural time")
print("  series model this guide describes; the estimate depends on the control")
print("  staying unaffected and the pre-period relation continuing.")

Printed output:

Simulated data, seed 24; built-in effect: +4.0 outcome units per day
Question: mean change in the outcome over the 14 post days,
  relative to a model fit before the change?
Inputs: 60 pre days and 14 post days; outcome and a simulated control.
Method: least squares of outcome on intercept, time, control (pre days);
  predict the post days.
Pre-period RMSE: 2.05
Mean estimated post effect: 3.14 outcome units per day
Sum of estimated post effects: 43.93 outcome units
Built-in effect minus estimate: 0.86
Limit: a simple simulated regression, not the Bayesian structural time
  series model this guide describes; the estimate depends on the control
  staying unaffected and the pre-period relation continuing.

Result. The pre-period RMSE was 2.05. The mean estimated effect is 3.14 outcome units per day and the sum over the 14 post days is 43.93 outcome units, against the built-in 4.0 per day. The estimate is 0.86 below the built-in effect, and this script reports no interval, so it cannot say how far off a run should be expected to land.

Limit. This is one simulated run with one seed. It shows how an estimate is read next to a known built-in value, not how any real campaign or system behaves.

When to Use Causal Impact (And When Not To)

Choosing the right analytical approach matters. Causal impact can be applied where simpler methods would suffice, or where the method's assumptions are fundamentally violated. Here's a comparison of appropriate and inappropriate use cases.

Ideal Use Cases

Geographic interventions: When you launch a campaign in specific regions, states, or countries, causal impact can use unaffected regions as controls. For example, measuring the impact of a regional advertising campaign on sales by comparing treated regions to similar untreated ones.

Policy changes: Government regulations, internal policy shifts, or platform-wide updates that affect all users simultaneously. Since randomization is impossible, a causal impact model with good control series can be one defensible estimate of effect size, provided its assumptions are stated.

Post-hoc analysis: Events have already occurred, and you need to quantify their impact retroactively. Traditional experiments require prospective design, but causal impact works with historical data.

Infrastructure changes: Technical modifications affecting your entire system where you cannot create a holdout group. Examples include server migrations, algorithm updates, or security implementations.

When to Choose Alternative Methods

User-level interventions: If you can randomly assign users to treatment and control groups, A/B testing provides stronger causal inference with simpler interpretation. Causal impact is a second-best option when randomization is impossible.

Multiple simultaneous interventions: If several changes occur simultaneously, causal impact cannot disentangle their individual effects. You'll estimate a combined effect but cannot attribute it to specific changes.

Insufficient historical data: With few pre-intervention observations, or a pre-period that is short next to the post-period, the model has little to learn from and its predictions become very uncertain; there is no universal minimum, so check how well the model predicts a held-out stretch of the pre-period. Simple interrupted time series or difference-in-differences may be more appropriate.

Fundamental structural changes: If your business model changed dramatically before the intervention (new product lines, market entry, acquisitions), historical patterns are poor predictors of counterfactual outcomes.

If you can still hold out a random group before the campaign starts, do that instead: a holdout test measures incrementality directly.

New to the idea of incrementality? Start with what incremental impact actually means, in plain words and with no data. For the analyst version, with the numbers and the checks, see estimating causal lift from a randomized holdout.

Critical Data Requirements: Comparing Good vs Poor Setups

The quality of your causal impact analysis depends heavily on your data quality and structure. Here's a side-by-side comparison of what works and what fails.

Time Series Length: Give the Model Enough History

Wrong approach: Using equal pre-intervention and post-intervention periods, or worse, having more post-intervention data than pre-intervention data. This starves the model of information needed to learn patterns.

Right approach: Give the model a pre-period several times longer than the post-period; there is no fixed ratio, and any ratio you pick is a rule of thumb, not a statistical requirement. If you want to measure a 4-week campaign effect, a pre-period of a few months gives the model something to learn from. Then check the model's accuracy on a held-out stretch of the pre-period.

A longer pre-period helps the Bayesian structural time series model learn seasonality, trends, and relationships with control variables before making predictions. A short pre-period is a common mistake in causal impact analysis.

Control Variable Selection: Avoiding Contamination

Wrong approach: Including variables that might be affected by your intervention. For example, when measuring a pricing change's impact on revenue, using website traffic as a control variable. The price change likely affects traffic, creating a contaminated control that underestimates the true effect.

Right approach: Select control variables that correlate with your outcome but cannot be influenced by the intervention. For a regional marketing campaign, use sales from unaffected regions, competitor metrics, or macroeconomic indicators. The key test: could this variable plausibly change because of my intervention?

Good control variables share common external factors with your treated unit but remain isolated from the intervention's direct and indirect effects. Think of them as parallel universes experiencing the same background conditions but not your specific treatment.

Data Frequency and Granularity

Wrong approach: Using daily data when you only have a few weeks of pre-intervention history, or aggregating to monthly data when you have years of daily observations. Mismatched frequency to data availability produces either noisy or overly smoothed models.

Right approach: Match your data frequency to both your available history and the intervention's expected time scale. For interventions lasting weeks, daily or weekly data works well if you have months of history. For interventions lasting months, weekly or monthly data is appropriate. The model needs enough observations to detect patterns at your chosen frequency.

Setting Up Your Causal Impact Analysis: Step-by-Step

Implementing causal impact analysis requires careful planning and execution. Here's a practical workflow comparing naive and rigorous approaches at each stage.

Step 1: Define Your Intervention Period Precisely

Ambiguity about when your intervention started and ended introduces measurement error that biases results. Many analysts use the official launch date, but the actual effect may begin earlier or later.

Best practice: Use the earliest point when the intervention could have affected outcomes. For a marketing campaign with a Monday launch, if materials were distributed Friday, use Friday as the intervention start. Being conservative about the intervention boundary prevents contamination of your pre-period.

Step 2: Explore and Validate Your Pre-Period Data

Before running the model, examine your pre-intervention data for issues that will undermine predictions.

Check for structural breaks: Plot your outcome variable over time. Do you see sudden level shifts, trend changes, or unusual spikes in the pre-period? If major changes occurred shortly before your intervention, your model may struggle to separate those effects from your treatment.

Assess stationarity: While the Bayesian structural time series model can handle some non-stationarity, extreme trends or explosive growth patterns can reduce prediction accuracy. If your metric grew very fast in the pre-period, projecting counterfactual growth is highly uncertain.

Identify missing data: Gaps in your time series force the model to interpolate, reducing confidence in predictions. Either fill gaps with appropriate methods or exclude periods with missing data from your analysis.

Step 3: Select and Validate Control Variables

Control variables improve prediction accuracy but only if chosen carefully.

Correlation test: Your control variables should correlate with the outcome during the pre-period. Calculate correlation coefficients and visualize relationships. There is no universal correlation cutoff for a useful control; judge each one by how well the model predicts a held-out stretch of the pre-period with it included.

Placebo test: Run a "fake" intervention on a pre-intervention date. If your model detects significant effects when none should exist, your controls are inadequate or your data has unmodeled patterns.

Exclusion restriction: Document why each control variable cannot be affected by your intervention. This qualitative assessment is as important as statistical tests.

Step 4: Configure Model Parameters

The Bayesian structural time series model underlying causal impact has several components you can enable or disable.

Trend: Include a local linear trend component if your outcome metric shows sustained upward or downward movement in the pre-period. Disable for stationary metrics.

Seasonality: Add seasonal components matching your data's natural cycles. Match the seasonal component to the frequency of your data: daily data can include day-of-week effects, weekly data week-of-year effects, and monthly data month-of-year effects, and you need at least a couple of full cycles in the pre-period to estimate them. Don't add seasonality to data without recurring patterns.

Regression on controls: Include them if you have valid control variables. The model learns the relationship between controls and outcome, then uses control variable values in the post-period to improve counterfactual predictions.

Example configuration in R using the CausalImpact package:

library(CausalImpact)

# `data` must hold the outcome and any controls (see the worked example)

# Define pre and post periods
pre.period <- c(1, 100)
post.period <- c(101, 130)

# Run analysis with custom model
impact <- CausalImpact(data,
                       pre.period,
                       post.period,
                       model.args = list(
                         nseasons = 7,  # Weekly seasonality
                         season.duration = 1,
                         dynamic.regression = TRUE
                       ))

# View results
summary(impact)
plot(impact)

Interpreting Causal Impact Output: Avoiding Misinterpretation

The causal impact model produces several outputs that are frequently misunderstood. Here's how to read them correctly.

Posterior Predictions and Confidence Intervals

The model generates a posterior distribution of what would have happened without intervention. The point estimate (usually the mean) is your counterfactual prediction, and the credible interval (often 95%) represents uncertainty.

Common mistake: Treating the point estimate as truth. The counterfactual is itself uncertain. Wide credible intervals mean you're less confident about what would have happened, making it harder to detect effects.

Correct interpretation: Focus on the probability that the effect is positive (or negative). The model reports "Posterior probability of a causal effect." Higher values mean stronger evidence of an effect; any cutoff is a convention, not a test, so read the probability together with the credible interval.

Absolute vs Relative Effects

Causal impact reports both absolute effects (e.g., +1,000 sales) and relative effects (e.g., +15% increase). Use both for complete understanding.

Absolute effect: This shows the change in the outcome's own units. State the unit and time span, then compare the change with the cost of the event. Report the relative effect too, so readers can see the change as a share of the usual outcome.

Relative effect: Answers "how meaningful is this change?" A +1,000 unit increase means something very different if baseline sales were 1,000 (100% increase) versus 100,000 (1% increase). Relative effects enable comparison across different scales.

Cumulative vs Point-in-Time Effects

The model shows how effects evolved over the intervention period. Don't just read the cumulative summary.

Time series plot: Examine the day-by-day or week-by-week comparison of actual vs predicted values. Did the effect appear immediately or build over time? Did it fade or strengthen? These dynamics inform whether you should continue, modify, or end the intervention.

Delayed effects: Some interventions, like brand awareness campaigns, may show lagged effects. If you end measurement too soon, you'll underestimate total impact. If effects are still growing at the end of your post-period, extend measurement or note that cumulative effects are lower bounds.

Worked Example: A Simulated Campaign With a Known Lift

This example is simulated, so the true answer is known and the method can be checked against it. A treated region and two control regions share one demand driver (a slow trend, a weekly cycle and random drift). A true +8% lift is added to the treated region for the last 28 of 208 days. The numbers are synthetic and invented for teaching; the script runs as written (statsmodels 0.14, seed 42) and the output below came from it.

import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.tsa.statespace.structural import UnobservedComponents

# Simulated daily revenue ($k): treated region and two control regions that share
# a common demand driver. A true +8% lift is added to the treated region from day 181.
rng = np.random.default_rng(42)
n_pre, n_post = 180, 28
n = n_pre + n_post
t = np.arange(n)
common = 100 + 0.05 * t + 6 * np.sin(2 * np.pi * t / 7) + np.cumsum(rng.normal(0, 0.6, n))
control_a = 0.8 * common + rng.normal(0, 2.0, n)
control_b = 1.1 * common + rng.normal(0, 2.0, n)
treated = 1.0 * common + rng.normal(0, 2.0, n)
treated[n_pre:] *= 1.08                       # the true effect: +8%

df = pd.DataFrame({"treated": treated, "control_a": control_a, "control_b": control_b})
pre, post = df.iloc[:n_pre], df.iloc[n_pre:]

# Naive: mean of the 28 days before the campaign vs the 28 days of the campaign
naive = post["treated"].mean() - df["treated"].iloc[n_pre - n_post:n_pre].mean()
print(f"naive before/after: {naive:+.1f}k per day")

# Counterfactual: local level + regression on the controls, fitted on the pre-period only
model = UnobservedComponents(pre["treated"], level="local level",
                             exog=pre[["control_a", "control_b"]]).fit(disp=False)
fc = model.get_forecast(steps=n_post, exog=post[["control_a", "control_b"]])
pred = fc.predicted_mean
lo, hi = fc.conf_int(alpha=0.05).T.values
effect = post["treated"].values - pred.values
print(f"estimated lift: {effect.mean():+.1f}k per day "
      f"({100 * effect.sum() / pred.sum():+.1f}% of the counterfactual)")
print(f"true lift: {(post['treated'].values - post['treated'].values / 1.08).mean():+.1f}k per day (+8.0%)")
print(f"days above the 95% band: {(post['treated'].values > hi).sum()} of {n_post}")

# Placebo: pretend the campaign began 28 days earlier, using only pre-period data
p_fit, p_test = pre.iloc[:n_pre - n_post], pre.iloc[n_pre - n_post:]
pm = UnobservedComponents(p_fit["treated"], level="local level",
                          exog=p_fit[["control_a", "control_b"]]).fit(disp=False)
pf = pm.get_forecast(steps=n_post, exog=p_test[["control_a", "control_b"]])
pe = p_test["treated"].values - pf.predicted_mean.values
print(f"placebo lift: {pe.mean():+.1f}k per day ({100 * pe.sum() / pf.predicted_mean.sum():+.1f}%)")

naive before/after: +11.9k per day
estimated lift: +8.8k per day (+8.4% of the counterfactual)
true lift: +8.4k per day (+8.0%)
days above the 95% band: 27 of 28
placebo lift: -0.4k per day (-0.4%)

What the output shows

The naive comparison overstates the effect. Comparing the 28 days before the campaign with the 28 days of it gives +11.9k per day, because, by construction of this simulation, the shared trend and drift keep moving while the campaign runs. The true lift is +8.4k per day.

The counterfactual model lands close to the truth. A local-level model with a regression on the two controls, fitted on the 180 pre-campaign days only, predicts what the treated region would have done. The gap between actual and predicted is +8.8k per day (+8.4%), against a true +8.0%. That is one simulated run, so treat the closeness as a demonstration, not a guarantee; a different seed gives a different error.

The placebo check came back flat. Pretending the campaign began 28 days earlier and refitting on the earlier data gives -0.4k per day (-0.4%), about zero. If the placebo had shown a lift of similar size to the real one, the controls would not be explaining the treated region well and the real estimate could not be trusted.

What this does not show. The controls here are unaffected by construction. In real data that is an assumption you have to argue for (see the exclusion restriction above), and a weak or contaminated control moves the estimate in ways the interval does not capture.

Best Practices for Reliable Causal Impact Analysis

Synthesizing lessons from common mistakes, here are essential best practices for trustworthy analysis.

Pre-Analysis Planning

Document your analysis plan before running any models. Specify your primary outcome metric, intervention period definition, control variables, and minimum detectable effect size. This prevents p-hacking and selective reporting.

Conduct a power analysis if possible. With limited pre-period data or high variance, you may be unable to detect realistic effect sizes. Knowing this in advance prevents misinterpreting null results as "no effect" when you simply lacked statistical power.

Model Diagnostics

Never skip diagnostic checks. Examine residual plots, autocorrelation functions, and posterior predictive checks. If diagnostics reveal problems, revise your model specification rather than reporting flawed results.

Check prediction quality on held-out pre-intervention data. Fit the model on the earlier part of your pre-period, then evaluate predictions on the later part. Poor out-of-sample prediction accuracy warns that post-intervention predictions are unreliable.

Sensitivity Analysis

Test how robust your conclusions are to analytical choices:

• Try different control variable combinations
• Vary the intervention start date by a few days
• Adjust the pre-period length
• Change seasonal component specifications

If your conclusion flips based on reasonable alternative choices, report this uncertainty rather than presenting a single definitive answer.

Clear Communication

When reporting results, always include:

• The exact intervention period
• Amount of pre and post-intervention data
• Control variables used and justification
• Both absolute and relative effect estimates
• Credible intervals, not just point estimates
• Posterior probability of effect
• Key limitations and assumptions

Visualize the entire time series with actual values, counterfactual predictions, and credible intervals. Non-technical stakeholders understand graphs better than tables of numbers.

Comparing Causal Impact to Related Techniques

Causal impact sits within a broader toolkit of causal inference methods. Understanding when to use alternatives improves your analytical decision-making.

Difference-in-Differences (DiD)

Similarity: Both methods use untreated units to construct counterfactuals and require parallel trends assumption.

Difference: DiD often uses simpler linear regression models and works with panel data (multiple units over time). Causal impact uses time series models and can work with a single treated unit.

When to choose DiD: You have panel data with multiple treated and control units, you want simpler interpretation, or you need to satisfy reviewers who prefer traditional econometric approaches.

When to choose causal impact: You have a single treated unit, complex time series dynamics, or need probabilistic uncertainty quantification from the Bayesian framework.

Synthetic Control Method

Similarity: Both construct counterfactuals using weighted combinations of control units.

Difference: Synthetic control explicitly creates a weighted average of control units that matches pre-intervention outcomes. Causal impact uses a regression-based approach within a Bayesian structural time series model.

When to choose synthetic control: You have multiple potential control units and want to transparently show which controls contribute to your counterfactual.

When to choose causal impact: You prefer the Bayesian uncertainty quantification, want to incorporate time series components like seasonality explicitly, or have natural control variables that don't need weighting.

Interrupted Time Series (ITS)

Similarity: Both analyze a single time series before and after an intervention.

Difference: Basic ITS uses simple regression models (often just linear trends), while causal impact uses flexible Bayesian structural time series.

When to choose ITS: You have limited data, need maximum transparency and simplicity, or work in fields where ITS is the established standard (like public health).

When to choose causal impact: You have enough data for a more flexible model, complex patterns requiring flexible time series components, or control variables to improve predictions.

Advanced Considerations and Limitations

Multiple Interventions

If you have several interventions over time, you can analyze them sequentially. Define each intervention period separately and ensure adequate gaps between interventions for clear attribution. Alternatively, combine small related interventions into a single analysis window.

Avoid analyzing overlapping interventions with a single causal impact model, as the method cannot disentangle simultaneous effects. Use more advanced techniques or design interventions to occur separately.

Time-Varying Effects

The standard causal impact model assumes constant treatment effects during the post-period. If you expect effects to grow or decay, examine the point-in-time effects plot rather than just the cumulative summary.

For complex time-varying dynamics, consider breaking the post-period into segments and analyzing each separately, or use extensions of the basic model that explicitly model effect evolution.

Spillover Effects

If your intervention might affect control units (for example, a marketing campaign in one region creating awareness in neighboring regions), your counterfactual is contaminated. The estimated effect will be a lower bound of the true effect.

Mitigate spillovers by choosing geographically or contextually distant controls, or acknowledge this limitation in your reporting. In some cases, spillover is itself interesting to measure by comparing nearby and distant controls.

Computational Considerations

Bayesian structural time series models use Markov Chain Monte Carlo (MCMC) sampling, which can be computationally intensive for very long time series or many control variables.

For faster iteration during model development, use shorter MCMC chains or subsample your data. Once you've settled on a specification, run full chains for final results.

Writing this up for publication? See statistical analysis for researchers. Citations and R code included.

Conclusion: Making Better Decisions Through Rigorous Analysis

Causal impact analysis is one way to estimate an intervention's effect when randomized experiments are impossible. It comes with assumptions that need checking. Common mistakes to avoid: insufficient pre-period data, contaminated control variables, ignored model diagnostics, and misinterpreted outputs.

By comparing naive approaches to best practices throughout this guide, you've learned not just how to run causal impact analysis, but how to run it correctly. Remember the critical success factors: give the model a pre-period several times longer than the post-period, select control variables the event cannot affect, check your model, and interpret results with appropriate uncertainty.

Done carefully, a time series estimate can be one input to the decision to continue or stop an intervention, next to cost and other evidence. The estimate is only as good as the model and its assumptions.

Start your next analysis with clear planning, proceed with careful validation, and report results with honest acknowledgment of limitations. This approach helps keep a causal impact analysis from only confirming existing beliefs.

Frequently Asked Questions

What is causal impact analysis?

Causal impact analysis estimates the effect of an intervention or event by comparing actual outcomes to a time series model's prediction of what might have happened without it. It uses time series data and, optionally, control variables. The estimate depends on the model and its assumptions.

When should I use causal impact instead of A/B testing?

Use causal impact when randomized experiments are impractical or impossible, such as evaluating marketing campaigns affecting entire regions, measuring policy changes, or analyzing events that have already occurred. A/B testing is preferred when you can randomly assign users to treatment and control groups before the intervention.

What are the most common mistakes in causal impact analysis?

The most common mistakes include using insufficient pre-intervention data (a pre-period that is short next to the intervention period), selecting control variables affected by the intervention, ignoring model diagnostics, misinterpreting statistical significance, and analyzing data with structural breaks in the pre-period. Validation and careful variable selection matter.

How much data do I need for causal impact analysis?

A pre-period several times longer than the post-period is a sensible target, but there is no universal minimum; check how well the model predicts a held-out stretch of the pre-period. If yearly seasonality matters, the pre-period should span several full yearly cycles where possible. More data can improve model accuracy and tighten intervals.

Can causal impact analysis handle multiple interventions?

Yes, but with limitations. You can analyze multiple sequential interventions by carefully defining intervention periods, but simultaneous interventions make it difficult to attribute effects. The best approach is to analyze one intervention at a time or use more advanced methods like synthetic control with multiple treatments.