Your shipping dashboard shows an average delivery time. Your satisfaction score sits where it sits. Nobody has shown that one drives the other, and when you ask whether faster shipping would improve ratings, you get hand-waving about "customer experience." This guide explains how to test the relationship with a regression of review scores on delivery time, how to read the output, how to look for a late-delivery threshold effect, and how to design the experiment that turns an association into a decision. It reports no results from any dataset.
The Research Question: Correlation or Causation?
An analysis of delivery time against customer satisfaction uses observational data. No one randomized half your orders to slow shipping and half to fast shipping, so you are measuring what naturally happened. That is correlation. It is valuable, because correlation tells you where to look, but it is not proof that changing delivery speed will change satisfaction.
The design question to answer before interpreting any regression: what confounds are lurking? If high-value orders get priority shipping, and high-value customers are more forgiving, your delivery coefficient will be biased. If product category drives both delivery complexity and review tendencies, you are measuring the wrong thing. OLS regression controls for observed confounders, but only if you include them in the model.
Setting Up the Data and the Model
You need order timestamps (purchased, delivered, estimated delivery), review scores (1 to 5 stars), and order status (delivered, shipped, canceled). Compute delivery_days = delivered_timestamp - purchased_timestamp, optionally bin deliveries into windows such as 0-5, 5-10, 10-20 and 20+ days, then regress review score on delivery days while controlling for order status and other confounders. The goal is to quantify the star penalty per day and to identify nonlinear effects such as late-delivery cliffs.
Sample Size and Statistical Power
Before you start, calculate the minimum detectable effect. The sample you need depends on the effect size you care about, the standard deviation of review scores, your significance level (commonly 0.05) and your target power (commonly 80%). Smaller effects need larger samples.
If your dataset is small relative to the effect you hope to detect, the test is underpowered. You will either miss a real effect or overfit noise. A very large dataset is the opposite case: it can make even trivial effects statistically significant, so judge practical importance by the size of the coefficient, not only the p-value.
Delivery Time Distribution: What to Look For
Start with a histogram of delivery days. Delivery times are usually right-skewed, with most orders clustering in a middle window and a long tail of very slow orders. Three things to check:
- The core window and the tail: The mode shows your core delivery performance. The tail is not just late. It often represents operational failures that will dominate your negative reviews.
- Power for subgroups: The distribution determines how many observations you have in each range. If you want to compare 3-day shipping with 5-day shipping, you need plenty of orders in that range, or a proper A/B test with randomized assignment.
- Sample selection in the tail: Orders that take very long may be fundamentally different (international shipping, custom products, processing errors). If those orders would have gotten bad reviews regardless of delivery speed, including them inflates your delivery coefficient. Controlling for order status partly addresses this, but observational data cannot fully separate "slow because complex" from "slow because inefficient."
Average Review Score by Delivery Bin
A bar chart of average review score by delivery bin is a first look at a dose-response relationship. If scores fall step by step as the delivery window grows, the relationship is graduated and measurable. If the last bin drops much faster than the earlier ones, that suggests a threshold effect, where customers tolerate some delivery variance but satisfaction collapses past a point.
Run the rigor check: is the relationship causal, or is it selection bias? If customers who order bulky furniture (slow shipping) are inherently harder to please than customers who order phone cases (fast shipping), you are measuring product mix, not delivery speed. Binned averages cannot answer this. That is why you need regression with controls.
Reading the OLS Regression Coefficients
The regression output gives a coefficient on delivery_days: the average change in review score per extra day in transit, holding the controls constant. Read it in four parts.
The order status coefficients are controls. Compare their magnitude to the delivery-days effect: if order status explained most of the variance, delivery speed would matter little.
Now the causation caveat: this coefficient measures association conditional on observed controls. If unobserved factors (customer patience, product expectations, regional logistics quality) correlate with both delivery time and review scores, the estimate is still biased. To get a causal estimate, you would need to randomize delivery speed. That is an A/B test: assign orders randomly to standard versus expedited shipping, measure the review score difference, and calculate the treatment effect. Observational regression tells you "this is worth testing." The experiment tells you "this is worth rolling out."
Illustrative example with made-up numbers: how to run the experiment
Hypothesis: Suppose a regression coefficient of -0.25 stars per day. Then reducing delivery time by 3 days would be expected to raise average review score by about 0.75 stars (3 days x 0.25 stars per day) if the association were causal.
Design: Randomly assign half of new orders to expedited shipping and half to standard shipping. Stratify randomization by product category and order value to ensure balance.
Sample size: Suppose the review score standard deviation is 1.2 and you want 80% power at alpha=0.05 to detect a 0.75-star difference. The two-sample formula n = 2 x (Zalpha/2 + Zbeta)2 x SD2 / delta2 gives about 2 x 7.85 x 1.44 / 0.5625, or roughly 40 orders per arm. Plan for more to allow subgroup analysis and attrition. A smaller true effect needs far more orders.
Metrics: Primary outcome is average review score. Secondary outcomes: share of 5-star reviews, share of 1-star reviews, repeat purchase rate within 90 days.
Runtime: Long enough to accumulate the planned number of orders with reviews. Reviews arrive some time after delivery, so allow for that lag.
Decision rule: Decide in advance the improvement that would justify a rollout. If the expedited arm clears it, calculate cost per incremental star and compare to customer lifetime value uplift before rolling out.
Late Delivery Rate vs 1-Star Rate
This comparison answers a different question: what happens when you miss the estimated delivery date? Build a binary indicator late = (delivered_at > estimated_delivery_at), then compare the late delivery rate and the 1-star review rate across order statuses or segments.
The design insight: late delivery may not just continue a linear per-day penalty. It can trigger a shift in the review distribution. A customer who receives an order on day 9 when you promised day 7 may not give 3.5 stars. They may give 1 star. If that threshold effect is present, it has major implications for how you set delivery estimates. You may be better off promising a longer window and meeting it than promising a short one and missing it. The linear model says longer delivery hurts, but a broken promise can hurt more.
From a testing perspective, measure two outcomes separately: (1) average review score, and (2) share of reviews that are 1-star. The first captures linear effects. The second captures threshold failures. If an intervention reduces average delivery time but increases delivery variability, you might win on the first metric and lose on the second.
Confounders You Must Control For
Controlling for order status is a start. Real-world e-commerce has many confounders that will bias your delivery coefficient if ignored:
- Product category: Some categories ship fast and get reviewed harshly, others ship slow and get reviewed for quality, not speed. Without a category control you mix very different products.
- Order value: High-value orders might get white-glove delivery or extra scrutiny. If expensive orders ship faster and customers are more forgiving, your delivery coefficient is biased.
- Geographic region: Urban orders arrive faster than rural ones. If rural customers have different satisfaction baselines, region confounds delivery time.
- Time of year: Holiday seasons slow everything and customers expect it. Pooling peak and off-peak deliveries can overestimate the delivery penalty unless you control for seasonality.
- Seller performance: Marketplaces with multiple sellers have seller-level variation. A seller with great products and poor logistics will show a spurious delivery-satisfaction correlation. Include seller fixed effects.
How do you know if a confounder matters? Run the regression with and without the control. If the delivery_days coefficient changes by more than about 10%, you have confounding. If it does not change, the confounder is either balanced or irrelevant. Do not include controls "just in case," because every control costs degrees of freedom and adds complexity. But test the ones that matter.
How to Interpret Your Regression Output
When you run this analysis on your own data, you get a delivery_days coefficient, a standard error, a p-value and a confidence interval. Here is how to interpret each piece:
Coefficient: The point estimate of the star penalty per day. Multiply by the gap between your delivery window and a competitor's to get the total rating impact of that gap.
Standard error: This tells you precision. Larger samples give smaller standard errors. Standard error determines your confidence interval width.
P-value: This tests the null hypothesis that delivery time has no effect (coefficient = 0). A p-value does not tell you effect size. You can have a very small p-value with a trivially small coefficient (statistically significant but practically irrelevant). Look at the coefficient first, the p-value second.
Confidence interval: This gives the range of plausible true effects. A narrow interval means a precise estimate. A very wide interval, for example one that runs from a large effect to nearly zero, means you need more data.
R-squared: This tells you what fraction of review score variance the model explains. The rest is product quality, customer expectations, support interactions and so on. Do not expect a high R-squared from a delivery-only model. You are isolating one driver, not predicting reviews perfectly.
When the Linear Model Breaks Down
OLS regression assumes a linear relationship: each additional day has the same marginal effect. That may hold in the middle of the delivery range where most data lives. A late delivery cliff would violate linearity at the extremes, and you cannot extrapolate a coefficient fitted on the middle range to very long deliveries.
If you suspect nonlinearity, test it. Add a squared term: review_score ~ delivery_days + delivery_days² + controls. If the squared term is significant and negative, you have accelerating penalties. Alternatively, include the binary late indicator: regress review_score on delivery_days + late + controls. The late coefficient captures the threshold jump beyond the linear trend.
Another breakdown: interaction effects. If delivery time matters more for high-value orders (because expectations are higher), include an interaction: delivery_days × order_value. If the interaction coefficient is significant, the delivery penalty scales with order size. That tells you where to prioritize faster shipping.
Finally, watch for heteroscedasticity. If review variance increases with delivery time (long deliveries have more unpredictable satisfaction), OLS standard errors are biased. Use robust standard errors (HC3, or clustered by seller) to fix this. Most stats packages offer this as a one-line option. If your confidence intervals change materially when you switch to robust standard errors, you had heteroscedasticity bias.
The Experiment You Should Run Next
Observational regression can point to delivery time as a candidate driver. The next step is an experiment that measures causation and lets you calculate ROI. Here is the protocol:
Treatment: Expedited shipping for the treatment group and standard shipping for the control group, with a gap large enough to produce a detectable effect given your regression estimate.
Randomization: Assign orders randomly at checkout. Use a hash function on order_id to ensure reproducibility and prevent contamination. Stratify by product category and order value to ensure balance, which reduces variance and increases power.
Sample size: Run a power analysis from the effect size you want to detect and the standard deviation of review scores. Add a margin for subgroup analysis.
Metrics: Primary outcome is average review score. Secondary outcomes: (1) share of 5-star reviews, (2) share of 1-star reviews, (3) repeat purchase rate at 90 days, (4) Net Promoter Score. Track cost per expedited delivery to calculate ROI.
Analysis: Run a two-sample t-test comparing treatment and control review scores. Calculate the difference, 95% confidence interval and p-value. If the effect is significant and large enough to matter, estimate incremental revenue from higher ratings (use historical data on rating to conversion rate) and compare to expedited shipping cost.
One critical point: intention-to-treat analysis. Some expedited shipments will miss the target, and some standard shipments will arrive early. Analyze by assigned group, not actual delivery time. If you analyze by actual delivery, you reintroduce selection bias (orders that arrived fast might be easier orders). Intention-to-treat preserves randomization and gives you the causal effect of offering expedited shipping, which is what you would actually deploy.
Red Flags That Your Test Is Broken
Imbalanced groups: If the treatment group has noticeably more high-value orders than control, your randomization failed. Check category and value distributions before launching. If imbalanced, re-randomize or use stratified assignment.
Crossover contamination: If customers can see both treatment and control offers (for example, multiple tabs), they will choose the faster option and your groups are not independent. Randomize at customer level, not order level.
Peeking at results: If you check p-values daily and stop the test when p < 0.05, you inflate the false positive rate well above 5%. Pre-commit to a sample size, run to completion, analyze once.
Post-hoc segmentation: If you slice data many ways looking for a winning segment, you are p-hacking. Pre-register subgroups (for example, "test separately for electronics and apparel") or do not claim significance.
Why This Matters for Customer Lifetime Value
A fraction of a star can sound small. But review scores are public, persistent and heavily weighted by marketplace algorithms and by customers who filter by rating. A modest rating gain can change visibility on marketplaces that use rating filters.
Illustrative example with made-up numbers: suppose preventing a 1-star review raises repeat purchase probability by 5 percentage points, and a repeat customer is worth $300 in lifetime value. Then each prevented 1-star review is worth about 0.05 x $300 = $15 in expectation. Suppose expedited shipping costs $8 more per order and prevents 15 1-star reviews per 100 orders. You would pay $800 to save 15 x $15 = $225 in lifetime value. In this invented case the upgrade does not pay for itself on 1-star prevention alone, and you would need the broader rating improvement and conversion lift to justify it. These figures are made up to show the arithmetic. Substitute your own.
This is why you run the experiment. Observational regression gives you a candidate effect size. Unit economics gives you the dollar value. The experiment confirms causation and measures it precisely enough to make the investment decision with confidence.
Common Mistakes and How to Avoid Them
Mistake 1: Treating correlation as causation. A regression coefficient is an association. It does not prove that speeding up delivery will improve reviews. Unobserved confounders (product quality, customer patience, regional effects) might drive both. Solution: run a randomized experiment to measure the causal effect.
Mistake 2: Ignoring late delivery threshold effects. If you only look at the linear coefficient, you may miss a 1-star cliff when deliveries run late. Customers may not reduce ratings proportionally. Solution: separately measure the share of orders that miss the estimated delivery date and the share of 1-star reviews in that segment.
Mistake 3: Using insufficient sample size. If your sample is small relative to the effect you want to detect, your test is underpowered. You will either miss a real effect (Type II error) or find a spurious one (overfitting noise). Solution: calculate required sample size before collecting data.
Mistake 4: Failing to control for confounders. If high-value orders get faster shipping and high-value customers rate more generously, your delivery coefficient conflates delivery speed with customer segment. Solution: include order value, product category and geographic region as controls. Run with and without controls to test sensitivity.
Mistake 5: Extrapolating beyond your data range. If most deliveries fall in a middle range, do not use the fitted coefficient to predict satisfaction for much faster or much slower deliveries. The relationship might be nonlinear at extremes. Solution: restrict predictions to the observed range or test nonlinearity with squared terms.
Mistake 6: Forgetting about delivery variance. Average delivery time is one metric. But if you promise 7 days and deliver anywhere from 5 to 10 days, you may get more 1-star reviews than a competitor who promises 10 days and delivers in 9 to 11. Solution: measure the standard deviation of delivery time and include "late delivery rate" as a separate metric.
The Analysis Workflow, Step by Step
The full workflow involves five steps:
Step 1: Data join. You need two tables: one with order timestamps (order_id, purchased_at, delivered_at, estimated_delivery_at, order_status) and one with review scores (order_id, review_score). Join on order_id, drop rows with missing delivery or review data, and compute delivery_days = (delivered_at - purchased_at).days.
Step 2: Binning and summary stats. Bin orders into delivery windows. For each bin, calculate average review score, standard deviation and count. This produces the "average review score by delivery bin" chart and shows whether the relationship is monotonic.
Step 3: OLS regression. Fit the model review_score ~ delivery_days + C(order_status) using ordinary least squares. Extract coefficients, standard errors, 95% confidence intervals and p-values. Check residuals for normality and homoscedasticity. If heteroscedasticity is detected, refit with robust standard errors.
Step 4: Late delivery analysis. Create a binary indicator late = (delivered_at > estimated_delivery_at). Calculate the late delivery rate and the 1-star review rate (share of reviews with score 1) for each order status. This shows whether threshold effects exist.
Step 5: Visualization and reporting. Produce a distribution histogram, a binned bar chart, a coefficient plot with error bars, and a late versus 1-star comparison chart, with statistical tables and interpretation notes.
You can replicate this workflow in R or Python with a short script. The hard part is not the code. It is knowing which metrics to calculate, which confounders to control for, and how to interpret the results without falling into the causation trap.
What to Do With Your Results
You have run the regression. You have a coefficient, a confidence interval and a p-value. Now what?
If the delivery_days coefficient is significant and large: Delivery time is a major satisfaction driver in your data. Calculate the cost per incremental improvement (for example, the cost to cut average delivery by a few days) and compare it to the value of the rating increase (conversion rate lift x customer LTV). If the economics look favorable, design an A/B test to confirm causation and measure the true treatment effect.
If the coefficient is significant but small: Delivery time matters, but it is not your primary satisfaction driver. Check R-squared. If it is low, most variance comes from other factors (product quality, support, price expectations). Run a broader regression including product category, order value and other controls. Identify the biggest drivers and prioritize those.
If the coefficient is not significant (p > 0.05): Either delivery time does not affect satisfaction, or your sample size is too small to detect the effect. Check your sample size against the minimum detectable effect. If you have a large sample and still see no effect, delivery time genuinely might not matter for your business (for example, if you sell custom products where customers expect long lead times).
If late delivery shows a threshold effect (the 1-star rate spikes): Focus on reducing delivery variance and late deliveries, not just average speed. Even if you cannot ship faster, you can promise more conservatively and hit estimates reliably. Test whether moving to a longer promised window reduces the 1-star rate despite longer average delivery.
Your turn
Bring your own data and the question you actually need answered.
MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai
Frequently Asked Questions
How do I measure what each delivery day costs in star rating?
Regress review score on delivery days with OLS, controlling for order status and other confounders. The delivery-days coefficient is the average change in review score per extra day, and its confidence interval shows how precise the estimate is. It is an association, not a proven causal effect. To measure the causal effect, randomize orders to expedited versus standard shipping and compare review scores.
Is there a cliff effect when deliveries run late?
It is worth testing for. A linear day-by-day penalty can coexist with a threshold effect when an order misses the promised date. Add a binary late indicator (delivered after the estimated date) to the regression and compare the share of 1-star reviews for late and on-time orders. If the late coefficient is large beyond the linear trend, missing the promise costs more than slow but predictable delivery, and delivery date accuracy matters as much as speed.
What sample size do I need to detect delivery time effects?
Run a power calculation before collecting data. It depends on the effect size you want to detect, the standard deviation of review scores, your significance level and your target power. Smaller effects and noisier outcomes need larger samples, and an experiment with subgroup analysis needs more orders per arm than a simple two-group comparison. Underpowered tests are worse than no tests, because you will either miss real effects or overfit noise.
Should I control for order value or product category?
Yes, if they are correlated with delivery time. If high-value orders get priority shipping, failing to control for order value will confound your estimate, because you will conflate customer segment with delivery speed. If furniture ships slower than electronics, failing to control for category mixes product expectations with delivery effects. Run the regression with and without controls. If the delivery coefficient changes by more than about 10%, you have confounding and must include controls.
Can I use this regression to predict the impact of faster shipping?
Observational regression tells you about association, not causation. To predict an intervention's impact, you need an experiment: randomize a sample of orders to expedited shipping, measure the review score difference versus control, and calculate ROI from the causal estimate. The regression tells you "this is worth testing." The experiment tells you "this is worth rolling out."