Can You Stop This Test Early? Peek Without Breaking It

How it works

Rows are ordered by arrival and the two arms' means are recomputed after every single observation. At each of those points the analysis evaluates a mixture sequential probability ratio test: with V the sampling variance of the effect estimate and a N(0, tau^2) mixture over the true effect, the mixture likelihood ratio against the null is Lambda = sqrt(V/(V+tau^2)) exp(delta^2 tau^2 / (2 V (V+tau^2))). Ville's inequality bounds the probability that Lambda ever reaches 1/alpha at alpha, which is what makes monitoring after every observation safe. Inverting the same boundary in the effect gives a closed-form confidence sequence, effect +/- sqrt(2 V (V+tau^2)/tau^2 log(sqrt((V+tau^2)/V)/alpha)), that covers the true effect at every sample size simultaneously. The always-valid p-value is the running minimum of 1/Lambda. Alongside it, an ordinary Welch two-sample t-test is computed on the same cumulative data at the same points, so the report can count how often the fixed-horizon test would have declared significance where the always-valid boundary did not. Futility is declared when the confidence sequence lies entirely inside tau, the effect the boundary is tuned to find; when the interval is wider than tau no boundary is reachable and the analysis reports that instead of a null result. All of it is implemented directly from the normal mixture in base R — no group-sequential package is involved.

Use it when the experiment is being watched while it runs, when you want the option to stop early on either a winner or futility, or when you need to know whether a flat-looking result is genuinely flat or merely underpowered.

Not when you can genuinely commit to one analysis at a sample size fixed in advance — the fixed-horizon test is more efficient and you should use it. Not for more than two arms in one file, not for repeated measures of the same person, and not as cover for a comparison chosen after the results were visible.

What data do you need?

One row per visitor: the variant seen, whether they converted, and when they arrived. The arrival column can be a date, a timestamp, or a plain sequence number:

Minimum 30 rows · Best with 1,000-10,000 rows across exactly 2 arms

What's in the report?

Standard-library analysis: can I stop this experiment early? Upload an ordered experiment log — the variant each row saw, what happened, and the date or sequence the observation arrived — and get inference that stays valid under continuous monitoring. The analysis re-tests after every single observation using a mixture sequential probability ratio test, and returns the always-valid confidence sequence over time, the current decision (stop for a winner, stop for futility, keep running, or not enough data for any boundary to be reachable), the boundary that was crossed and when, and the cumulative effect charted with both boundaries drawn over it. The naive fixed-horizon p-value is computed alongside at the same points, so the report can show — from your own data, not from an assertion — how many times it would have crossed 0.05 while the always-valid boundary did not. The report also states what the guarantee costs in power, in this experiment's own numbers.