Standard Incrementality
Executive Summary

Executive Summary

How much of the movement in Weekly Sales was incremental, against 12 holdout markets.

Markets
20
Treated Markets
8
Control Markets
12
Pre Periods
16
Post Periods
8
Incremental Lift
139.43
Lift CI Low
110.39
Lift CI High
168.47
Relative Lift (%)
11.7
Parallel Trends
SUPPORTED
The treated markets ran +139.43 in Weekly Sales per market-period against what the controls say they would have done anyway — a relative lift of +11.7% on a counterfactual level of 1,190.63 (95% interval 110.39 to 168.47, p < 0.001). The interval excludes zero, so the lift is distinguishable from market-to-market noise. The integrity check behind it: parallel trends SUPPORTED — no detectable pre-intervention divergence (-0.416 per period, p = 0.526). This is a failure to detect a violation, not proof that trends were parallel. Standard errors are clustered on the 20 markets, so the interval reflects market-to-market variation rather than treating each market-period as independent. Difference-in-differences also assumes no other shock hit only the treated markets during the post window: a regional promotion, a competitor exit, a supply problem or weather that landed on one arm alone would be counted as lift.
Suggested Interpretation

The treated markets delivered +139.43 in incremental Weekly Sales per market-period against 12 holdout controls—a relative lift of 11.7% on a counterfactual baseline of 1,190.63 (95% CI: 110.39 to 168.47, p < 0.001). The interval excludes zero, so the lift is distinguishable from market-to-market noise. Parallel trends SUPPORTED: no detectable pre-intervention divergence (−0.416 per period, p = 0.526). This is a failure to detect a violation, not proof of trend parallelism. The estimate rests on the assumption that no other shock—regional promotion, competitor exit, supply disruption, or localized weather—hit only the treated markets during the post window. Standard errors are clustered on the 20 markets, reflecting market-to-market variation rather than treating each observation as independent.

Overview

Analysis Overview

Difference-in-differences on 20 markets in 'Market' — 8 treated against 12 held back.

N Units20
N Treated Units8
N Control Units12
N Pre Periods16
N Post Periods8
Standard Errorsunit-clustered (CR1)
Suggested Interpretation

The incremental effect is estimated as a difference of differences: the change in Weekly Sales from before to after in the 8 treated markets, minus the change over the same periods in the 12 control markets. The controls serve as the counterfactual—they carry whatever else moved Weekly Sales in that window, which is why this is a stronger causal claim than projecting a single series' own pre-period trend forward. The specification is two-way fixed effects on Weekly Sales, with a fixed effect for every market and every period, absorbing persistent market size differences and shocks common to all markets before the treatment effect is read off. Standard errors are unit-clustered (CR1) on the 20 markets, so the interval reflects market-to-market variation rather than treating each market-period as independent. The method assumes no concurrent shock hit only the treated markets during the post window—a regional promotion, competitor exit, supply problem, or localized weather would be counted as lift.

Data Preparation

Data Preparation

How the raw rows became a treated-versus-control panel.

Initial Rows480
Final Rows480
Rows Dropped0
Rows Aggregated0
N Periods24
Balanced PanelTrue
Suggested Interpretation

480 rows loaded with no missing values. Each market appeared once per period, so no aggregation was needed. In 'Treatment Group', 'yes' was read as treated; 'no' as control. In 'Period', 'post' was read as after intervention; 'pre' as before. This yields 8 treated and 12 control markets over 16 pre-periods and 8 post-periods, with the post window opening at 2024-04-22. The panel is balanced: 20 markets × 24 periods = 480 observations, every cell present. 'Week' parsed as dates, so the charts use the real calendar timeline.

Visualization

Treated vs Control Over Time

Mean Weekly Sales per period for each arm.

Suggested Interpretation

Mean Weekly Sales per period for the 8 treated markets and 12 control markets. Before the intervention opened at 2024-04-22, the two lines moved with no detectable divergence—the gap between them could be any size, but it should not be widening or narrowing. The pre-period differential trend was −0.416 per period, which supports the parallel-trends assumption. After the intervention, the treated arm stepped away from the control arm, with the post-period estimates ranging from 124.22 to 152.10 in incremental Weekly Sales. The visual is the foundation of the method: the change in the gap between the arms after intervention, against what the gap was doing before.

Visualization

Effect by Period (Event Study)

Per-period treated-minus-control effect on Weekly Sales, against the last pre-intervention period.

Suggested Interpretation

Each point is the treated-versus-control difference in Weekly Sales for one period, measured against the last pre-intervention period (2024-04-15), which is fixed at zero by construction. None of the 15 pre-intervention estimates misses zero—the 95% intervals all contain zero—which is what a clean design looks like. After the intervention opened at 2024-04-22, the estimates step away from zero and remain elevated: 2024-04-22 at 140.06 (CI: 95.85 to 184.27), 2024-04-29 at 137.37 (CI: 104.81 to 169.93), 2024-05-06 at 143.48 (CI: 101.10 to 185.86), and 2024-06-10 at 152.10 (CI: 110.99 to 193.21). The step-and-stay pattern is evidence the effect is real and persistent, not a gradual drift that began before the opening.

Data Table

The Difference-in-Differences 2x2

Cell means before and after, for each arm, and their differences.

GroupBeforeAfterChange
Treated (8 markets)11961330134.1
Control (12 markets)11171112-5.315
Difference (treated minus control)78.67218.1139.4
Suggested Interpretation

The four cell means the method rests on. Treated markets moved +134.12 in Weekly Sales from before to after; control markets moved −5.31 over the same periods. The difference of those two changes is +139.43—the raw difference-in-differences. The headline estimate of +139.43 comes from the fixed-effects regression, which matches this arithmetic on a balanced panel but attaches a standard error to it. The control arm's near-flat trajectory (−5.31) means most of what happened in the treated markets was incremental, not counterfactual drift. That difference is exactly what the incrementality question asks.

Data Table

Parallel Trends Check

Were the arms already diverging before the intervention?

CheckValueInterpretation
Differential pre-intervention trend per period (treated minus control), in Weekly Sales-0.416Fitted on the 16 pre-intervention periods only: how fast the treated markets were pulling away from the controls BEFORE anything was switched on. Difference-in-differences assumes this is zero.
95% interval on that differential trend-1.764 to 0.932Standard errors are unit-clustered (CR1). An interval comfortably containing zero is what a clean design looks like.
p-value of the differential pre-trend0.526Below 0.05 the arms were already diverging and the headline estimate cannot be read as causal.
Pre-intervention periods available16More pre-intervention periods make this check sharper; 16 are available here.
Pre-intervention event-study estimates whose 95% interval excludes zero0 of 15Each pre-intervention period is also estimated separately against the last pre-period; estimates that miss zero are direct evidence against parallel trends. None of them miss zero here.
Bias in the estimate if that pre-trend simply continued-1.87The pre-trend projected across the post window is -1.87, or 1.3% of the estimated effect — that is how much of the headline +139.43 could be pre-existing drift rather than the intervention.
VerdictSUPPORTEDSUPPORTED — no detectable pre-intervention divergence (-0.416 per period, p = 0.526). This is a failure to detect a violation, not proof that trends were parallel
Suggested Interpretation

Over the 16 pre-intervention periods, there is no detectable divergence between the arms (−0.416 per period, p = 0.526), with a 95% interval of −1.764 to 0.932. An interval comfortably containing zero is what a clean design looks like. None of the 15 pre-intervention event-study estimates misses zero, providing direct corroboration. If the pre-trend had continued into the post window, it would have injected −1.87 of bias—or 1.3% of the estimated effect. The verdict is SUPPORTED: a failure to detect a violation, not proof of trend parallelism. With 16 pre-periods available, the test has reasonable power. Credibility depends on the assumption that no other shock—regional promotion, competitor exit, supply disruption, localized weather—landed on the treated markets alone during the post window.

Methodology

Methodology

Statistical methodology and diagnostics for Incrementality Test (Geo Lift)

Statistical Method

Incrementality Test (Geo Lift)

Standard-library analysis: how much of the lift was actually incremental. You turned something on in some markets and deliberately held others back — this measures the difference-in-differences between the two arms, so the untreated markets carry whatever would have happened anyway. Two-way fixed-effects estimate with unit-clustered 95% intervals, a parallel-trends integrity check that says out loud when the estimate is not credible, an event study showing when the effect started, and the 2x2 of cell means underneath it all.

Data
N = 480 observations
Assumptions
  • Parallel trends: without the intervention, treated and control units would have moved the same way — tested explicitly and reported
  • The treatment flag is constant within a unit and the after flag is constant within a period
  • No other shock hit only the treated units during the post window
  • Treatment of one unit does not spill over into the control units
Limitations
  • Difference-in-differences assumes no concurrent shock landed on the treated units alone — a regional promotion, competitor exit or weather event on one arm is counted as lift
  • If parallel trends are violated the estimate is not causal, and no amount of extra modelling in this tool fixes it — the design has to change
  • With fewer than 10 units the cluster-robust standard error is unreliable, so classical errors are used and the 95% interval is optimistic
  • Spillover from treated into control units (shared media, cross-market shopping) biases the estimate toward zero
Software & Citation
MCP Analytics · mcpanalytics.ai
Code Appendix

Analysis Code

Complete R source code for this analysis

Incrementality Test — How Much Lift Was Actually Incremental?

Geo-lift / holdout incrementality measurement by difference-in-differences on a panel of units (markets, stores, regions). Some units were treated (advertising switched on, a change rolled out); the rest were deliberately held back. The untreated units supply the counterfactual: what the treated units would have done anyway.

Why This Method?

A single series' own past can only tell you what it was trending toward. Untreated control units also absorb whatever else happened that period — a demand swing, a competitor, a holiday — because it hit both arms. That is why difference-in-differences with real holdouts is a far stronger causal claim than a before/after comparison of one series against its own pre-period trend.

What This Analysis Covers

  • The difference-in-differences estimate from a two-way fixed-effects

regression, with unit-clustered standard errors and a 95% interval

  • The parallel-trends check: were the arms already diverging before the

intervention? If so, the estimate is not credible and the report says so

  • An event study: per-period effect relative to the last pre-period
  • The 2x2 of cell means the whole method rests on

Standard Library

Platform standard-library module (LAT-1441): runs on ANY panel via the semantic mapping {unit, period, metric, treated, post}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Core Analysis Pipeline

Step 1: Every mapped column must be present

need <- c("unit", "period", "metric", "treated", "post")
  human <- c(unit = unit_name, period = period_name, metric = metric_name,
             treated = treated_name, post = post_name)
  missing_keys <- setdiff(need, names(df))
  if (length(missing_keys) > 0) {
    stop(sprintf(
      "Incrementality testing needs a market column(&#x27;%s'), a period column ('%s'), a numeric outcome ('%s'), a treatment-group flag ('%s') and a before/after flag ('%s'). %s could not be found in the data.",
      unit_name, period_name, metric_name, treated_name, post_name,
      paste0("&#x27;", paste(human[missing_keys], collapse = "', '"), "'")))
  }

Step 2: Coerce the outcome (95% rule) — refuse if it is not numeric

v <- df$metric
  if (!is.numeric(v)) {
    conv <- suppressWarnings(as.numeric(as.character(v)))
    n_orig <- sum(!is.na(v) & nzchar(trimws(as.character(v))))
    if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
      v <- conv
    } else {
      stop(sprintf(
        "&#x27;%s' is not numeric — measuring incremental lift needs a numeric outcome (sales, conversions, revenue).",
        metric_name))
    }
  }
  v <- as.numeric(v)

Step 3: Build the timeline. Dates are used when the period column

parses as dates; otherwise the distinct period labels are ordered (numerically when they are numbers) and laid out on an evenly spaced synthetic timeline purely so the charts have a time axis. Which of the two happened is stated in the report.

per_raw <- trimws(as.character(df$period))
  per_raw[is.na(df$period) | !nzchar(per_raw)] <- NA_character_
  pd <- parse_dates_robust(per_raw)
  n_nonblank <- sum(!is.na(per_raw))
  if (n_nonblank == 0) {
    stop(sprintf("&#x27;%s' is empty — there are no periods to compare before and after the intervention.",
                 period_name))
  }
  period_is_date <- sum(!is.na(pd)) >= 0.95 * n_nonblank

Step 4: Binarize the two flags, then keep complete rows only

unit_chr <- trimws(as.character(df$unit))
  unit_chr[is.na(df$unit) | !nzchar(unit_chr)] <- NA_character_

  tb_treated <- function(clean, levs) {
    list(level = levs[length(levs)],
         reason = sprintf(
           "neither label is a standard yes/no word, so the alphabetically later label &#x27;%s' was taken as the treated group",
           levs[length(levs)]))
  }
  tb_post <- function(clean, levs) {
    ord <- if (period_is_date) as.numeric(pd) else
      as.numeric(factor(per_raw, levels = sort(unique(per_raw))))
    mean_pos <- tapply(ord, clean, function(z) mean(z, na.rm = TRUE))
    mean_pos <- mean_pos[is.finite(mean_pos)]
    if (length(mean_pos) < 2) {
      return(list(level = levs[length(levs)],
                  reason = sprintf("the periods could not be ordered, so the alphabetically later label &#x27;%s' was taken as the after-intervention side",
                                   levs[length(levs)])))
    }
    lv <- names(mean_pos)[which.max(mean_pos)]
    list(level = lv,
         reason = sprintf("neither label is a standard before/after word, and &#x27;%s' falls later on the timeline on average", lv))
  }

  tr <- binarize_flag(
    df$treated, treated_name, "^(1|true|yes|y|t|treated|treatment|test|on|exposed)$",
    tb_treated,
    paste0("&#x27;", treated_name,
           "&#x27; holds only one value ('%s') — a holdout incrementality test needs BOTH treated markets and untreated control markets. Without controls there is no counterfactual, and the honest answer is that incremental lift cannot be measured from this data."))
  po <- binarize_flag(
    df$post, post_name, "^(1|true|yes|y|t|post|after|during|on)$",
    tb_post,
    paste0("&#x27;", post_name,
           "&#x27; holds only one value ('%s') — the analysis needs periods from BOTH before and after the intervention started."))

  treated01 <- ifelse(tr$clean == tr$on_level, 1L, 0L)
  post01    <- ifelse(po$clean == po$on_level, 1L, 0L)

  keep <- !is.na(unit_chr) & !is.na(per_raw) & !is.na(v) &
    !is.na(treated01) & !is.na(post01)
  n_dropped_rows <- initial_rows - sum(keep)
  w <- data.frame(
    unit = unit_chr[keep],
    period_label = per_raw[keep],
    period_date = if (period_is_date) pd[keep] else as.Date(NA),
    metric = v[keep],
    treated01 = treated01[keep],
    post01 = post01[keep],
    stringsAsFactors = FALSE)
  if (nrow(w) == 0) {
    stop(sprintf(
      "No rows survived cleaning — every row is missing at least one of &#x27;%s', '%s', '%s', '%s' or '%s'.",
      unit_name, period_name, metric_name, treated_name, post_name))
  }

Step 5: Order the periods and lay out the timeline

if (period_is_date) {
    ord_levels <- sort(unique(w$period_date))
    w$tidx <- match(w$period_date, ord_levels)
    axis_dates <- ord_levels
    period_axis_note <- sprintf(
      "&#x27;%s' parsed as dates, so the charts use the real calendar timeline",
      period_name)
  } else {
    labs <- unique(w$period_label)
    num <- suppressWarnings(as.numeric(labs))
    ord_levels <- if (all(!is.na(num))) labs[order(num)] else sort(labs)
    w$tidx <- match(w$period_label, ord_levels)
    axis_dates <- as.Date("2000-01-01") + (seq_along(ord_levels) - 1L)
    period_axis_note <- sprintf(
      "&#x27;%s' does not hold dates, so its %d distinct labels were ordered and placed on an evenly spaced synthetic timeline; the chart's date ticks are placeholders for period order, not real calendar dates",
      period_name, length(ord_levels))
  }
  axis_iso <- format(axis_dates, "%Y-%m-%d")
  period_label_of <- ord_levels
  n_periods <- length(ord_levels)

Step 6: Integrity of the design — the treatment flag must be a

property of the unit, and the before/after flag a property of the period

bad_units <- names(which(tapply(w$treated01, w$unit,
                                  function(z) length(unique(z))) > 1))
  if (length(bad_units) > 0) {
    stop(sprintf(
      "%d %s in &#x27;%s' (%s) %s marked as treated in some rows and untreated in others. Difference-in-differences needs '%s' to be constant within each %s — one arm per market for the whole window.",
      length(bad_units), pl(length(bad_units), "market"), unit_name,
      paste(head(bad_units, 3), collapse = ", "),
      pl(length(bad_units), "is", "are"), treated_name, unit_name))
  }
  bad_periods <- names(which(tapply(w$post01, w$tidx,
                                    function(z) length(unique(z))) > 1))
  if (length(bad_periods) > 0) {
    bad_lab <- period_label_of[as.integer(bad_periods)]
    stop(sprintf(
      "%d %s in &#x27;%s' (%s) %s marked both before and after the intervention. '%s' must be constant within each period — every period falls entirely on one side of the start date.",
      length(bad_lab), pl(length(bad_lab), "period"), period_name,
      paste(head(bad_lab, 3), collapse = ", "),
      pl(length(bad_lab), "is", "are"), post_name))
  }

Step 7: Collapse duplicate unit-period rows by SUM (rows are read as

amounts that add up within one market-period), then count the design

n_before_agg <- nrow(w)
  w <- aggregate(metric ~ unit + tidx + treated01 + post01, data = w, FUN = sum)
  n_dup_rows <- n_before_agg - nrow(w)
  w$period_label <- period_label_of[w$tidx]
  w$period_iso <- axis_iso[w$tidx]
  w <- w[order(w$unit, w$tidx), , drop = FALSE]
  rownames(w) <- NULL

  units_all <- unique(w$unit)
  n_units <- length(units_all)
  unit_arm <- tapply(w$treated01, w$unit, max)
  n_treated_units <- sum(unit_arm == 1)
  n_control_units <- sum(unit_arm == 0)
  pre_idx  <- sort(unique(w$tidx[w$post01 == 0]))
  post_idx <- sort(unique(w$tidx[w$post01 == 1]))
  n_pre <- length(pre_idx); n_post <- length(post_idx)

Step 8: Refuse rather than degrade

if (n_control_units < 1) {
    stop(sprintf(
      "Every market in &#x27;%s' is in the treatment group ('%s' = '%s') — a holdout incrementality test needs untreated control markets to build the counterfactual.",
      unit_name, treated_name, tr$on_level))
  }
  if (n_treated_units < 1) {
    stop(sprintf(
      "No market in &#x27;%s' is marked as treated ('%s' = '%s') — there is nothing to measure the lift of.",
      unit_name, treated_name, tr$on_level))
  }
  if (n_pre < 2) {
    stop(sprintf(
      "Only %d pre-intervention %s in &#x27;%s' — at least 2 are needed, because with a single pre-period there is no way to check whether the arms were already diverging.",
      n_pre, pl(n_pre, "period"), period_name))
  }
  if (n_post < 1) {
    stop(sprintf(
      "No period in &#x27;%s' is marked as after the intervention ('%s' = '%s') — there is no post window to measure.",
      period_name, post_name, po$on_level))
  }
  if (n_units < 4) {
    stop(sprintf(
      "Only %d %s in &#x27;%s' (%d treated, %d control) — at least 4 are needed for a difference-in-differences estimate to carry any usable uncertainty. With fewer, the honest answer is that the lift cannot be separated from market-to-market noise.",
      n_units, pl(n_units, "market"), unit_name, n_treated_units,
      n_control_units))
  }
  n_obs <- nrow(w)
  balanced <- n_obs == n_units * n_periods

Step 9: Two-way fixed-effects DiD. The interaction of treated and

post IS the difference-in-differences estimate; market and period fixed effects absorb the two main effects, so the interaction is fitted directly to keep the design matrix full rank.

w$did <- w$treated01 * w$post01
  fit_did <- stats::lm(metric ~ did + factor(unit) + factor(period_label),
                       data = w)
  if (is.na(coef(fit_did)[["did"]])) {
    stop(sprintf(
      "The treated-by-after term could not be estimated: it is perfectly explained by the market and period effects. Check that treated markets in &#x27;%s' are observed in BOTH the before and after windows of '%s'.",
      unit_name, post_name))
  }
  n_clusters <- n_units
  few_clusters <- n_clusters < 10
  V_did <- cluster_vcov(fit_did, w$unit)
  cluster_ok <- !is.null(V_did) && !few_clusters
  inf_did <- coef_inference(fit_did, "did", V_did, n_clusters, cluster_ok)
  se_basis <- inf_did$basis

Step 10: Relative lift is the absolute effect over the

counterfactual level (observed treated-post mean minus the effect).

mean_treated_post <- mean(w$metric[w$treated01 == 1 & w$post01 == 1])
  cf_level <- mean_treated_post - inf_did$est
  rel_ok <- is.finite(cf_level) && abs(cf_level) > 1e-9
  rel_scale <- if (rel_ok) 100 / abs(cf_level) else NA_real_
  rel_est <- if (rel_ok) inf_did$est * rel_scale else NA_real_
  rel_lo  <- if (rel_ok) inf_did$lo * rel_scale else NA_real_
  rel_hi  <- if (rel_ok) inf_did$hi * rel_scale else NA_real_

Step 11: THE integrity gate — parallel trends. Fit the pre-period

only and ask whether the treated arm was already moving differently.

pre_w <- w[w$post01 == 0, , drop = FALSE]
  pre_w$treat_t <- pre_w$treated01 * pre_w$tidx
  fit_pt <- tryCatch(
    stats::lm(metric ~ treat_t + factor(unit) + factor(period_label),
              data = pre_w),
    error = function(e) NULL)
  if (!is.null(fit_pt) && !is.na(coef(fit_pt)[["treat_t"]])) {
    V_pt <- cluster_vcov(fit_pt, pre_w$unit)
    inf_pt <- coef_inference(fit_pt, "treat_t", V_pt, n_clusters, cluster_ok)
  } else {
    inf_pt <- list(est = NA_real_, se = NA_real_, df = NA_real_, p = NA_real_,
                   lo = NA_real_, hi = NA_real_, basis = "not estimable")
  }
  pt_ok <- is.finite(inf_pt$est) && is.finite(inf_pt$p)

If the measured pre-trend simply carried on, this is the bias it would inject into the DiD estimate over the post window.

mean_post_gap <- mean(post_idx) - max(pre_idx)
  pt_bias <- if (pt_ok) inf_pt$est * mean_post_gap else NA_real_
  pt_bias_share <- if (pt_ok && abs(inf_did$est) > 1e-9)
    100 * abs(pt_bias) / abs(inf_did$est) else NA_real_
  pt_verdict <- if (!pt_ok) "unassessable" else if (inf_pt$p < 0.05) "violated" else
    if (inf_pt$p < 0.10) "borderline" else "supported"

Step 12: Event study — per-period effect against the last

pre-intervention period, which is normalised to zero by construction.

ref_idx <- max(pre_idx)
  es_terms <- setdiff(seq_len(n_periods), ref_idx)
  es_params <- n_units + 2 * n_periods
  es_method <- "regression"
  es_rows <- NULL
  if (es_params <= 500 && length(es_terms) > 0) {
    es_names <- paste0("es", es_terms)
    for (i in seq_along(es_terms)) {
      w[[es_names[i]]] <- as.integer(w$treated01 == 1 & w$tidx == es_terms[i])
    }
    fml <- stats::as.formula(paste0(
      "metric ~ ", paste(es_names, collapse = " + "),
      " + factor(unit) + factor(period_label)"))
    fit_es <- tryCatch(stats::lm(fml, data = w), error = function(e) NULL)
    if (!is.null(fit_es)) {
      V_es <- cluster_vcov(fit_es, w$unit)
      es_rows <- do.call(rbind, lapply(seq_along(es_terms), function(i) {
        ii <- coef_inference(fit_es, es_names[i], V_es, n_clusters, cluster_ok)
        data.frame(tidx = es_terms[i], effect = ii$est,
                   ci_low = ii$lo, ci_high = ii$hi,
                   stringsAsFactors = FALSE)
      }))
    }
  }
  if (is.null(es_rows)) {

How many PRE-period estimates are significantly non-zero — the visual half of the parallel-trends evidence.

pre_es <- es_rows[es_rows$tidx %in% setdiff(pre_idx, ref_idx), , drop = FALSE]
  pt_n_bad_pre <- if (nrow(pre_es) == 0) 0L else
    as.integer(sum(pre_es$ci_low > 0 | pre_es$ci_high < 0, na.rm = TRUE))
  pt_n_pre_est <- nrow(pre_es)

  event_study_df <- data.frame(
    period = axis_iso[es_rows$tidx],
    effect = round(es_rows$effect, 4),
    ci_high = round(es_rows$ci_high, 4),
    ci_low = round(es_rows$ci_low, 4),
    stringsAsFactors = FALSE)

Step 13: The picture people actually want — mean outcome per period

for each arm.

arm_means_df <- do.call(rbind, lapply(seq_len(n_periods), function(k) {
    do.call(rbind, lapply(c(1L, 0L), function(a) {
      sel <- w$tidx == k & w$treated01 == a
      if (!any(sel)) return(NULL)
      data.frame(period = axis_iso[k],
                 mean_metric = round(mean(w$metric[sel]), 4),
                 arm = if (a == 1L) "Treated" else "Control",
                 stringsAsFactors = FALSE)
    }))
  }))
  if (nrow(arm_means_df) > 2000) {
    keep_k <- sort(unique(round(seq(1, n_periods, length.out = 1000))))
    arm_means_df <- arm_means_df[arm_means_df$period %in% axis_iso[keep_k], ,
                                 drop = FALSE]
  }
  rownames(arm_means_df) <- NULL

Step 14: The 2x2 the whole method rests on

cell <- function(a, p) {
    sel <- w$treated01 == a & w$post01 == p
    if (!any(sel)) NA_real_ else mean(w$metric[sel])
  }
  t_pre <- cell(1, 0); t_post <- cell(1, 1)
  c_pre <- cell(0, 0); c_post <- cell(0, 1)
  raw_did <- (t_post - t_pre) - (c_post - c_pre)
  did_cells_df <- data.frame(
    group = c(sprintf("Treated(%d %s)", n_treated_units,
                      pl(n_treated_units, "market")),
              sprintf("Control(%d %s)", n_control_units,
                      pl(n_control_units, "market")),
              "Difference(treated minus control)"),
    before = round(c(t_pre, c_pre, t_pre - c_pre), 4),
    after = round(c(t_post, c_post, t_post - c_post), 4),
    change = round(c(t_post - t_pre, c_post - c_pre, raw_did), 4),
    stringsAsFactors = FALSE)

Step 15: Round ONCE — the same numbers feed tables and prose

did_r  <- round(inf_did$est, 2)
  did_lo <- round(inf_did$lo, 2); did_hi <- round(inf_did$hi, 2)
  did_p  <- inf_did$p
  rel_r  <- if (is.na(rel_est)) NA_real_ else round(rel_est, 1)
  rel_lo_r <- if (is.na(rel_lo)) NA_real_ else round(rel_lo, 1)
  rel_hi_r <- if (is.na(rel_hi)) NA_real_ else round(rel_hi, 1)
  raw_did_r <- round(raw_did, 2)
  pt_slope <- if (pt_ok) round(inf_pt$est, 3) else NA_real_
  pt_lo <- if (pt_ok) round(inf_pt$lo, 3) else NA_real_
  pt_hi <- if (pt_ok) round(inf_pt$hi, 3) else NA_real_
  pt_bias_r <- if (pt_ok) round(pt_bias, 2) else NA_real_
  pt_bias_share_r <- if (is.na(pt_bias_share)) NA_real_ else round(pt_bias_share, 1)
  cf_level_r <- round(cf_level, 2)
  did_significant <- is.finite(did_lo) && is.finite(did_hi) &&
    (did_lo > 0 || did_hi < 0)
  direction_word <- if (is.finite(did_r) && did_r >= 0) "above" else "below"

Step 16: The parallel-trends diagnostic table

pt_verdict_sentence <- switch(
    pt_verdict,
    violated = sprintf(
      "VIOLATED — the arms were already diverging by %s per period before the intervention(%s), so the difference-in-differences estimate is not a credible causal number",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    borderline = sprintf(
      "BORDERLINE — the pre-period divergence of %s per period is not conclusive either way(%s); treat the estimate with caution",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    supported = sprintf(
      "SUPPORTED — no detectable pre-intervention divergence(%s per period, %s). This is a failure to detect a violation, not proof that trends were parallel",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    "UNASSESSABLE — the pre-period trend term could not be estimated from this panel")

  parallel_trends_df <- data.frame(
    check = c(
      sprintf("Differential pre-intervention trend per period(treated minus control), in %s", metric_name),
      "95% interval on that differential trend",
      "p-value of the differential pre-trend",
      "Pre-intervention periods available",
      "Pre-intervention event-study estimates whose 95% interval excludes zero",
      "Bias in the estimate if that pre-trend simply continued",
      "Verdict"),
    value = c(
      if (pt_ok) fmt_signed(pt_slope, 3) else "not estimable",
      if (pt_ok) paste0(fmt_n(pt_lo, 3), " to ", fmt_n(pt_hi, 3)) else "not estimable",
      if (pt_ok) fmt_p(inf_pt$p) else "not estimable",
      as.character(n_pre),
      sprintf("%d of %d", pt_n_bad_pre, pt_n_pre_est),
      if (pt_ok) fmt_signed(pt_bias_r) else "not estimable",
      toupper(pt_verdict)),
    interpretation = c(
      sprintf("Fitted on the %d pre-intervention %s only: how fast the treated markets were pulling away from the controls BEFORE anything was switched on. Difference-in-differences assumes this is zero.",
              n_pre, pl(n_pre, "period")),
      sprintf("Standard errors are %s. An interval comfortably containing zero is what a clean design looks like.",
              inf_pt$basis),
      "Below 0.05 the arms were already diverging and the headline estimate cannot be read as causal.",
      sprintf("More pre-intervention periods make this check sharper; %d %s available here.",
              n_pre, pl(n_pre, "is", "are")),
      sprintf("Each pre-intervention period is also estimated separately against the last pre-period; estimates that miss zero are direct evidence against parallel trends. %s",
              if (pt_n_bad_pre > 0)
                sprintf("%d of them %s zero here.", pt_n_bad_pre,
                        pl(pt_n_bad_pre, "misses", "miss"))
              else "None of them miss zero here."),
      if (pt_ok) sprintf(
        "The pre-trend projected across the post window is %s%s — that is how much of the headline %s could be pre-existing drift rather than the intervention.",
        fmt_signed(pt_bias_r),
        if (!is.na(pt_bias_share_r)) sprintf(", or %s%% of the estimated effect",
                                             fmt_n(pt_bias_share_r, 1)) else "",
        fmt_signed(did_r))
      else "The projection could not be computed because the pre-trend term is not estimable.",
      pt_verdict_sentence),
    stringsAsFactors = FALSE)

Step 17: Metrics + the one-paragraph computed answer

metrics <- list(
    `Markets`             = n_units,
    `Treated Markets`     = n_treated_units,
    `Control Markets`     = n_control_units,
    `Pre Periods`         = n_pre,
    `Post Periods`        = n_post,
    `Incremental Lift`    = did_r,
    `Lift CI Low`         = did_lo,
    `Lift CI High`        = did_hi,
    `Relative Lift(%)`   = rel_r,
    `Parallel Trends`     = toupper(pt_verdict)
  )

  lift_phrase <- paste0(
    fmt_signed(did_r), " in ", metric_name, " per market-period(95% interval ",
    fmt_n(did_lo), " to ", fmt_n(did_hi), ")",
    if (!is.na(rel_r)) paste0(", a relative lift of ", fmt_signed(rel_r, 1),
                              "% against a counterfactual level of ",
                              fmt_n(cf_level_r)) else "")
  answer <- if (pt_verdict == "violated") {
    paste0(
      "The parallel-trends assumption FAILS on this panel: treated and control markets in &#x27;",
      unit_name, "&#x27; were already diverging by ", fmt_signed(pt_slope, 3),
      " per period before the intervention(", fmt_p_phrase(inf_pt$p),
      "), so the difference-in-differences estimate of ", lift_phrase,
      " should NOT be read as incremental lift — an unknown part of it is pre-existing drift(projected bias ",
      fmt_signed(pt_bias_r),
      "). Fix the design before trusting a number: add more pre-intervention periods, choose control markets that tracked the treated ones before the change, or use a synthetic-control weighting of the controls.")
  } else {
    paste0(
      "Difference-in-differences across ", n_units, " markets(",
      n_treated_units, " treated, ", n_control_units, " held back) over ",
      n_pre, " pre and ", n_post, " post ", pl(n_post, "period"),
      ": the treated markets ran ", lift_phrase, " ", direction_word,
      " what the control markets say they would have done anyway. The interval ",
      if (did_significant) "excludes" else "includes",
      " zero, so the lift is ",
      if (did_significant) "distinguishable from" else "not distinguishable from",
      " noise. Parallel trends: ", toupper(pt_verdict), " (",
      fmt_p_phrase(inf_pt$p), " on the pre-period differential trend). ",
      "This assumes nothing else hit only the treated markets at the same time.")
  }

  json_output <- list(
    answer = answer,
    cards = lapply(
      c("tldr", "overview", "preprocessing", "arm_trends", "event_study",
        "did_table", "parallel_trends"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = n_obs,
    rows_removed = max(0, initial_rows - n_obs),
    n_dropped_rows = n_dropped_rows, n_dup_rows = n_dup_rows,
    unit_name = unit_name, period_name = period_name,
    metric_name = metric_name, treated_name = treated_name,
    post_name = post_name,
    treated_on = tr$on_level, treated_off = tr$off_level,
    treated_reason = tr$reason,
    post_on = po$on_level, post_off = po$off_level, post_reason = po$reason,
    period_is_date = period_is_date, period_axis_note = period_axis_note,
    n_units = n_units, n_treated_units = n_treated_units,
    n_control_units = n_control_units, n_periods = n_periods,
    n_pre = n_pre, n_post = n_post, n_obs = n_obs, balanced = balanced,
    first_post_label = period_label_of[min(post_idx)],
    ref_period_label = period_label_of[ref_idx],
    did_r = did_r, did_lo = did_lo, did_hi = did_hi, did_p = did_p,
    did_significant = did_significant, direction_word = direction_word,
    rel_r = rel_r, rel_lo = rel_lo_r, rel_hi = rel_hi_r, rel_ok = rel_ok,
    cf_level_r = cf_level_r, raw_did_r = raw_did_r,
    se_basis = se_basis, cluster_ok = cluster_ok, n_clusters = n_clusters,
    few_clusters = few_clusters,
    pt_slope = pt_slope, pt_lo = pt_lo, pt_hi = pt_hi, pt_p = inf_pt$p,
    pt_verdict = pt_verdict, pt_verdict_sentence = pt_verdict_sentence,
    pt_bias_r = pt_bias_r, pt_bias_share_r = pt_bias_share_r,
    pt_n_bad_pre = pt_n_bad_pre, pt_n_pre_est = pt_n_pre_est,
    es_method = es_method,
    arm_means_df = arm_means_df, event_study_df = event_study_df,
    did_cells_df = did_cells_df, parallel_trends_df = parallel_trends_df,
    metrics = metrics, json_output = json_output
  )
}

This is the exact R code executed to produce the analysis above. Run it with the same dataset to reproduce these results.

Your data has more stories to tell. Run any analysis on your own data — validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No Signup Sign Up Free
Something wrong? Report it and we'll fix it — or get your credits back
Download PDF · Print · Share ·Refresh this analysis ·Edit this report

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing