Anomaly detection

The method guide

The practical guide behind this tool: when it applies, how to read the output, and the traps that make it say the wrong thing.

Read the guide →

A simulated example, with the code and its output

The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.

Question: If a few rows are planted that do not fit the usual pattern, how often does anomaly detection flag them, and how often does it flag rows on clean data?

Inputs: Simulated data. Each dataset has 500 rows and 3 columns (a, b, c). Column a is standard normal. Column b is 0.8 times a plus noise, so a and b move together (correlation about 0.8). Column c is standard normal and independent. 200 seeds: seeds 0 to 199 make clean datasets, seeds 1000 to 1199 make datasets with planted rows. Built in, so these are expected by construction: in each planted dataset 30 rows are changed, in 6 groups of 5. Three groups are single-column spikes, where column c is pushed up by 3, 4 and 6 standard deviations (SD). Three groups are combinations, where a is set to +s and b to -s for s = 1.5, 2.0 and 3.0 SD; for s = 1.5 each value alone looks ordinary, but the pair breaks the a-b pattern. Typed thresholds, which are our settings and not results: flag a row if its robust z-score is above 3.5 in any column, or if its robust distance squared is above 16.27 (the 99.9% point of a chi-square distribution with 3 degrees of freedom). No contamination rate is typed, so the number of flags is not set in advance.

Method: Anomaly detection, here in two parts. Part 1, robust z-score: for each column take the median (middle value) and the MAD (median absolute deviation, the middle size of the gaps from the median). The robust z-score is 0.6745 times (value minus median) divided by MAD. It is called robust because a few extreme rows barely move the median or the MAD, so they cannot hide themselves. Part 2, robust distance, for rows that are odd only in combination: fit a mean and a covariance (how columns move together) on the 75% of rows closest to the center, repeat 10 times, then measure each row's distance (Mahalanobis distance, a distance that accounts for how columns move together) and rescale it. A row is flagged if either part fires. The script runs this on 200 clean datasets to get the false-flag rate, and on 200 planted datasets to get the detection rate for each planted size. This is a common simple way to flag unusual rows; the tool's own method may differ.

Result: On clean data 0.16% of rows were flagged, and 52% of clean datasets had at least one flagged row. In the planted data, on average 0.59 of the 470 normal rows per dataset were flagged. A spike in column c of 3 SD was caught in 26.9% of datasets, 4 SD in 60.5% and 6 SD in 99.3%. For the pair rows, 1.5 SD was caught in 0.0%, 2.0 SD in 19.4% and 3.0 SD in 100.0%. In the dataset from seed 1000, 15 rows were flagged and 12 of the 30 planted rows were among them. Each value of the 1.5 SD pair looks ordinary on its own, so a rule with these typed thresholds never flags it. The detection rates come from the planted sizes and the typed thresholds, so they describe this setup, not what the tool will catch in your data.

Limit: This example has one clean pattern (normal columns, one correlation) and a fixed pair of thresholds, so the detection rates would change with other data shapes, other thresholds, or heavy-tailed columns. It does not show how the tool picks its columns or thresholds, or how it explains why a row stands out.

Python script (numpy and scipy only):

import numpy as np
from scipy.stats import chi2
N, K, SEEDS = 500, 5, 200            # rows, planted rows per size, seeds
Z_CUT = 3.5                           # typed rule: flag if robust z > 3.5
D_CUT = chi2.ppf(0.999, 3)            # typed rule: flag if robust distance^2 > this
SPIKES = [3.0, 4.0, 6.0]              # planted: column c pushed up by this many SDs
COMBOS = [1.5, 2.0, 3.0]              # planted: a up by s SDs, b down by s SDs
def make(rng, plant):
    a = rng.normal(size=N); b = 0.8 * a + 0.6 * rng.normal(size=N); c = rng.normal(size=N)
    X = np.c_[a, b, c]; lab = {}; i = 0
    if plant:
        for s in SPIKES:
            X[i:i+K, 2] += s; lab[('spike', s)] = range(i, i+K); i += K
        for s in COMBOS:
            X[i:i+K, 0] = s; X[i:i+K, 1] = -s; lab[('combo', s)] = range(i, i+K); i += K
    return X, lab
def flags(X):
    med = np.median(X, 0); mad = np.median(np.abs(X - med), 0)
    rz = 0.6745 * (X - med) / mad                       # robust z-score per column
    f1 = (np.abs(rz) > Z_CUT).any(1)
    keep = np.ones(len(X), bool)                        # robust distance: refit on the 75% closest
    for _ in range(10):
        m = X[keep].mean(0); S = np.cov(X[keep].T)
        d2 = np.einsum('ij,jk,ik->i', X - m, np.linalg.inv(S), X - m)
        keep = d2 <= np.quantile(d2, 0.75)
    d2 = d2 / chi2.ppf(0.75, 3)                         # the fit used the closest 75%, so rescale
    f2 = d2 > D_CUT
    return f1 | f2, f1, f2
hit = {k: [] for k in [('spike', s) for s in SPIKES] + [('combo', s) for s in COMBOS]}
clean_rate, clean_any, other = [], [], []
for seed in range(SEEDS):
    X, _ = make(np.random.default_rng(seed), False)
    f, _, _ = flags(X); clean_rate.append(f.mean()); clean_any.append(f.any())
    X, lab = make(np.random.default_rng(1000 + seed), True)
    f, f1, f2 = flags(X)
    for k, r in lab.items(): hit[k].append(f[list(r)].mean())
    other.append(f[K*6:].sum())
print("Simulated data. rows =", N, "| planted rows per size =", K, "| seeds =", SEEDS)
print("Rules: robust z > %.1f in any column, or robust distance^2 > %.2f (chi2 99.9%%, 3 df)" % (Z_CUT, D_CUT))
print("Clean data: share of rows flagged = %.2f%%; datasets with at least 1 flag = %.0f%%" % (100*np.mean(clean_rate), 100*np.mean(clean_any)))
print("Planted data: normal rows flagged per dataset, mean = %.2f of %d" % (np.mean(other), N-6*K))
for (kind, s), v in hit.items():
    lab_txt = ("spike in column c of +%.1f SD" % s) if kind == 'spike' else ("combo a=+%.1f, b=-%.1f SD" % (s, s))
    print("Detection rate, %-30s %.1f%%" % (lab_txt, 100*np.mean(v)))
X, lab = make(np.random.default_rng(1000), True); f, f1, f2 = flags(X)
print("Seed 1000 example: flagged rows =", int(f.sum()), "| planted rows =", 6*K, "| planted rows caught =", int(f[:6*K].sum()))

Printed output from running the code above:

Simulated data. rows = 500 | planted rows per size = 5 | seeds = 200
Rules: robust z > 3.5 in any column, or robust distance^2 > 16.27 (chi2 99.9%, 3 df)
Clean data: share of rows flagged = 0.16%; datasets with at least 1 flag = 52%
Planted data: normal rows flagged per dataset, mean = 0.59 of 470
Detection rate, spike in column c of +3.0 SD   26.9%
Detection rate, spike in column c of +4.0 SD   60.5%
Detection rate, spike in column c of +6.0 SD   99.3%
Detection rate, combo a=+1.5, b=-1.5 SD        0.0%
Detection rate, combo a=+2.0, b=-2.0 SD        19.4%
Detection rate, combo a=+3.0, b=-3.0 SD        100.0%
Seed 1000 example: flagged rows = 15 | planted rows = 30 | planted rows caught = 12

What's in the report?

Which rows do not fit the usual pattern, including in combination, and what makes each stand out.