Market basket

The method, in full

The practical guide behind this tool: when it applies, how to read the output, and the traps that make it say the wrong thing.

Read the guide →

A simulated example, with the code and its output

The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.

Question: If two pairs of items are built to sell together and all other pairs are unrelated, do support, confidence, lift and a test adjusted for many pairs pick out the two real pairs and drop the chance ones?

Inputs: Simulated baskets, not real data. Seed 42 makes 2000 baskets over 12 items, each item with its own chance of being in a basket. Two links are built in: half of the bread baskets also get butter, and 40% of the chips baskets also get salsa. No other pair is linked. That gives 66 pairs to test.

Method: As in the tool's linked sample report: for each pair, support is the share of baskets holding both, confidence is the share of baskets with the first item that also hold the second, and lift is support divided by what it would be if the two were unrelated, so 1 means no link. Each pair gets a chi-square test on its 2 by 2 table (continuity corrected), and the p-values are adjusted across all 66 pairs by Benjamini-Hochberg. Pairs are kept at support of at least 1% and confidence of at least 20%, then ranked by lift.

Result: The two built-in pairs came out on top: chips and salsa with lift 2.98 and confidence 0.44, and bread and butter with lift 2.51 and confidence 0.54, both with adjusted q far below 0.05. Before adjustment 6 of the 66 pairs had p below 0.05; after Benjamini-Hochberg only the 2 built-in pairs did. Butter and eggs looked linked at p = 0.011 but its adjusted q was 0.19, a chance result.

Limit: Items here are independent apart from two strong, planted links in a large sample; real baskets have seasonal items, store effects and weak links that may not reach the adjusted cut-off, and a link between two items says nothing about whether one causes the other.

Python script (numpy and scipy only):

import numpy as np
from itertools import combinations
from scipy.stats import chi2_contingency
# Simulated baskets. 12 items, 2000 baskets. Each item is in a basket with its own chance.
# Built in: bread and butter go together, chips and salsa go together. Nothing else is linked.
rng = np.random.default_rng(42)
items = ["bread", "butter", "chips", "salsa", "milk", "eggs", "apples", "coffee", "tea", "rice", "pasta", "soap"]
p = [0.30, 0.10, 0.20, 0.08, 0.35, 0.25, 0.20, 0.15, 0.10, 0.12, 0.15, 0.05]
n = 2000
X = rng.random((n, len(items))) < np.array(p)
X[:, 1] |= X[:, 0] & (rng.random(n) < 0.50)    # half of bread baskets also get butter
X[:, 3] |= X[:, 2] & (rng.random(n) < 0.40)    # 40% of chips baskets also get salsa
rows = []
for i, j in combinations(range(len(items)), 2):
    a, b = X[:, i], X[:, j]
    both = (a & b).sum()
    t = [[both, (a & ~b).sum()], [(~a & b).sum(), (~a & ~b).sum()]]
    pv = chi2_contingency(t, correction=True)[1]
    sup = both / n; conf = both / a.sum(); lift = sup / (a.mean() * b.mean())
    rows.append((items[i], items[j], sup, conf, lift, pv))
m = len(rows)
order = np.argsort([r[5] for r in rows])
q = np.empty(m); prev = 1.0
for rank in range(m, 0, -1):                     # Benjamini-Hochberg step-up
    k = order[rank - 1]; prev = min(prev, rows[k][5] * m / rank); q[k] = prev
print("baskets %d, items %d, pairs tested %d" % (n, len(items), m))
keep = [k for k in range(m) if rows[k][2] >= 0.01 and rows[k][3] >= 0.20]
keep.sort(key=lambda k: -rows[k][4])
print("pair             support  conf   lift   raw p     BH q")
for k in keep[:5]:
    a, b, s, c, l, pv = rows[k]
    print("%-16s %6.3f  %5.2f  %5.2f  %8.2g  %8.2g" % (a + " > " + b, s, c, l, pv, q[k]))
print("pairs with raw p < 0.05: %d" % sum(r[5] < 0.05 for r in rows))
print("pairs with BH q < 0.05: %d" % sum(q < 0.05))

Printed output from running the code above:

baskets 2000, items 12, pairs tested 66
pair             support  conf   lift   raw p     BH q
chips > salsa     0.084   0.44   2.98     2e-70   6.7e-69
bread > butter    0.156   0.54   2.51  1.1e-111  7.4e-110
butter > eggs     0.065   0.30   1.19     0.011      0.19
bread > apples    0.066   0.23   1.09      0.23      0.96
chips > eggs      0.051   0.27   1.08      0.36      0.96
pairs with raw p < 0.05: 6
pairs with BH q < 0.05: 2

What's in the report?

Which items sell together: each pair of items with its support, confidence and lift, tested with a chi-square and adjusted for testing many pairs (Benjamini-Hochberg in the sample report), then ranked.