One-Class SVM: A Method Guide to Novelty and Anomaly Detection

1. Introduction and Scope

1.1 The problem

Normal operation tends to produce plenty of data, while anomalous events are rare, poorly documented or new. Binary and multi-class classifiers need representative examples of every class, so they fit badly when the abnormal class cannot be listed in advance. This arises in fraud screening, equipment monitoring, security monitoring and quality control, among other places.

The one-class approach learns from normal examples only. A one-class SVM builds a boundary around the training distribution in a kernel-induced feature space, and points outside that boundary are flagged as novelty or anomalies.

1.2 Scope of this paper

This paper is an explanation of the method. It covers:

• The formulation of the one-class SVM and the role of the kernel
• The meaning of the nu and gamma parameters
• Novelty detection versus outlier detection
• Assumptions, pitfalls and how to check the output
• How to apply it, with a scikit-learn script run on simulated data and on one public dataset

It does not benchmark the method against other methods or describe any deployment. The one run on real data (Section 4.4) is a single public dataset, so it shows how the settings behave, not how well the method works in general. Where a claim depends on data and is not from that run, it is stated as a property of the method or as general guidance.

1.3 Terms used in this paper

Feature

One measured column of your data, such as order value or sensor temperature.

Kernel

A formula that scores how similar two rows are. It lets the model draw curved boundaries without building new columns by hand.

Feature space

The space the kernel implicitly maps rows into. A flat boundary there can be a curved boundary in the original data.

Hyperplane

A flat boundary: a line in two dimensions, a plane in three, and the same idea in more.

Support vectors

The training rows on or outside the boundary. They alone define where the boundary sits.

Decision function

The number the model gives each row: positive inside the boundary, negative outside. Lower means more unusual.

nu (ν)

A setting between 0 and 1 that controls how many training rows the model may leave outside the boundary.

gamma (γ)

A setting of the RBF kernel that controls how local the boundary is: small gamma gives a smooth outline, large gamma hugs individual rows.

2. Background

2.1 Where the method comes from

Support Vector Machines were introduced by Vapnik and colleagues, and the kernel trick allows them to work in high-dimensional feature spaces without computing the mapping explicitly. Schölkopf and co-authors extended the idea to a single class in a 1999 conference paper on novelty detection and a 2001 journal article (both listed in the references), proposing to separate the training data from the origin in feature space with a maximum-margin hyperplane. Tax and Duin proposed Support Vector Data Description (SVDD), which instead fits the smallest hypersphere around the data. With an RBF kernel, for which every point has the same norm in feature space, Schölkopf et al. (2001) show that the two formulations give the same solution; Tax and Duin (2004) describe the two solutions as comparable.

2.2 Novelty detection versus outlier detection

The two terms describe different situations, and the difference affects how you use the model.

• Novelty detection: the training set is assumed to be clean. The model is fit on it and then applied to new observations, asking whether each one looks like the training data. This is the setting a one-class SVM is designed for.
• Outlier detection: the training set itself may contain anomalies, and the goal is to find them within it. A one-class SVM can be used this way, but it is sensitive to contamination. The scikit-learn user guide says the one-class SVM "is known to be sensitive to outliers and thus does not perform very well for outlier detection", and methods built for the task, such as Isolation Forest or robust covariance estimation, are often considered first.

2.3 Why not a simpler method?

Simple rules usually look at one feature at a time. Z-score thresholds assume a roughly bell-shaped distribution, while interquartile-range fences are computed without assuming a shape, although how many points they flag still depends on the shape of the data. Distance-based and density-based methods such as nearest neighbors or local outlier factor can handle more complex shapes but depend on a distance measure, which can become less informative as the number of dimensions grows. A one-class SVM offers a flexible boundary defined by a subset of the training points. Its costs are the need to choose a kernel and parameters, sensitivity to feature scaling, and an anomaly score that is not a probability.

3. Method

3.1 The formulation

Given training points x₁ to xₙ from the normal class, a kernel k(x, x') corresponds to a feature map φ. The one-class SVM finds a weight vector w and an offset ρ so that most points satisfy w·φ(x) ≥ ρ, while the margin ρ/||w|| of the hyperplane from the origin is as large as possible. The decision function is f(x) = w·φ(x) - ρ, and the predicted label is its sign: positive inside the learned region and negative outside it. The value of f itself, before the sign is taken, is the score used to rank points from most to least unusual. The problem is solved in its dual form, where the data enter only through kernel values, and the solution depends only on the support vectors.

3.2 The nu parameter

Nu, between 0 and 1, replaces the usual penalty constant of an SVM. It has two exact interpretations in the standard formulation, proved by Schölkopf et al. (2001):

• it is an upper bound on the fraction of training points that lie outside the boundary, and
• it is a lower bound on the fraction of training points that are support vectors.

These are bounds, not targets: the fraction of training points outside the boundary is at most nu, and the fraction of support vectors is at least nu. As the sample grows, both approach nu, provided the training points are independent draws from a continuous distribution and the kernel is a smooth, non-constant one such as the Gaussian. Numerical solvers stop at a tolerance, so a fitted model can land slightly off a bound. On new data the flagged fraction can differ from nu in either direction. Nu is best chosen from how many flags can be reviewed, not left at a default.

Assumptions beside the result: the two bounds hold for the exact solution of the optimization problem; the "both approach nu" result also needs independent draws from a continuous distribution and a suitable kernel.

3.3 Kernels and gamma

Linear kernel: k(x, x') = x·x'. It is cheap and the boundary is a plain hyperplane, which is a reasonable choice for high-dimensional sparse data such as bag-of-words features. With a linear kernel the region is unbounded in some directions, so it can be a poor description of a compact cloud of normal points.

RBF kernel: k(x, x') = exp(-γ||x - x'||²). It can describe non-linear and non-convex regions. Gamma sets how local the influence of each point is. A small gamma gives a smooth, broad boundary. A large gamma gives a tight boundary that follows individual points and can overfit, to the point of drawing a small region around each training point. The scikit-learn documentation for OneClassSVM states that gamma="scale" (the default) uses 1 / (n_features × X.var()). After standardizing, X.var() is about 1, so this is roughly 1 / n_features: 1/30 = 0.0333 in the example in Section 4.4. Assumption: the RBF kernel treats every feature as equally important after scaling, so it relies on the scaling step below.

Polynomial kernel: k(x, x') = (x·x' + c)^d. It models interactions up to degree d, but the values can become very large or very small, so it needs scaled inputs and a deliberate choice of degree.

3.4 Data preparation

Feature scaling: kernels based on Euclidean distance are dominated by features with large numeric ranges. Standardize or min-max scale before fitting, and fit the scaler on the training data only.

Training set purity: the model assumes the training data are mostly normal. Anomalies in the training set pull the boundary outward and make the model less likely to flag similar points later. Screen the training data first, and keep nu high enough to absorb the contamination you expect.

Dimensionality: kernel methods tolerate many features, but irrelevant features still add noise to distances. Feature selection or dimensionality reduction such as PCA can help when many features carry no signal.

3.5 Checking the output

Because there are no labeled anomalies in training, evaluation takes extra effort. Useful checks (Section 4.4 runs the second and fourth on one public dataset; this paper does not run the first and third):

• Inspect the most strongly flagged points (the lowest decision-function values in scikit-learn) by hand and ask whether a domain expert would call them unusual.
• If some labeled anomalies exist, keep them aside and measure how well the anomaly score (the negated decision-function value, so that higher means more unusual) ranks them above normal points, for example with precision at k or the area under the ROC curve. A small labeled set gives a noisy estimate.
• Insert known synthetic anomalies into held-out normal data and see whether they are flagged.
• Look at the fraction of support vectors and the fraction flagged on held-out normal data, and compare both with nu.

4. Worked Examples

Sections 4.1 to 4.3 use arithmetic and simulated data to show how nu behaves. Section 4.4 runs the method on a public dataset.

4.1 What nu implies

Suppose we train a one-class SVM on 1,000 normal observations with nu = 0.05. By the bounds in Section 3.2, at most 0.05 × 1,000 = 50 training points can lie outside the boundary, and at least 50 of the 1,000 points will be support vectors. If we raise nu to 0.20, those figures become at most 200 points outside and at least 200 support vectors. So a larger nu gives a looser fit that treats more training points as boundary cases, and the model is likely, though not certain, to flag more new points.

4.2 A sketch in scikit-learn

Suppose the matrix X_normal holds the training rows and X_new holds rows to be scored:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import OneClassSVM

model = make_pipeline(
    StandardScaler(),
    OneClassSVM(kernel="rbf", gamma="scale", nu=0.05),
)
model.fit(X_normal)

labels = model.predict(X_new)            # +1 inside, -1 outside
scores = model.decision_function(X_new)  # higher means more normal
order = scores.argsort()                 # most unusual first

From here, review the first rows in order, adjust nu and gamma, and refit. Check the fraction flagged on a held-out sample of normal data, since it should be small if the model is not overfitting.

4.3 A run on simulated data

The script below uses only simulated data: 1,000 training points and 1,000 new points from the same two-dimensional normal distribution, plus 50 points from a shifted distribution. It fits the model at two values of nu and prints the rates it measures.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.svm import OneClassSVM

rng = np.random.default_rng(7)
X_train = rng.normal(size=(1000, 2))
X_new = rng.normal(size=(1000, 2))
X_far = rng.normal(loc=4.0, size=(50, 2))

scaler = StandardScaler().fit(X_train)
Z_train = scaler.transform(X_train)
Z_new = scaler.transform(X_new)
Z_far = scaler.transform(X_far)

print("nu   outside  support  new-flagged  far-flagged")
for nu in (0.05, 0.20):
    m = OneClassSVM(kernel="rbf", gamma="scale", nu=nu).fit(Z_train)
    out = (m.predict(Z_train) == -1).mean()
    sv = len(m.support_) / len(Z_train)
    new = (m.predict(Z_new) == -1).mean()
    far = (m.predict(Z_far) == -1).mean()
    row = (nu, out, sv, new, far)
    print("%.2f   %.3f    %.3f    %.3f        %.2f" % row)

nu   outside  support  new-flagged  far-flagged
0.05   0.054    0.066    0.062        1.00
0.20   0.199    0.207    0.202        1.00

We ran this with scikit-learn 1.7.1, NumPy 2.3.2 and Python 3.12; the second block is the printed output. Reading the output: the fraction of training points outside the boundary lands close to nu (0.054 against a bound of 0.05, which is a small numerical-tolerance overshoot of the kind noted in Section 3.2), and the fraction of support vectors sits slightly above nu. Fresh normal points are flagged at about the same rate. Raising nu to 0.20 flags roughly three times as many normal points, and the shifted points are flagged either way. This is one simulated run with one random seed, so the exact figures will differ with other data.

4.4 A run on a public dataset: Breast Cancer Wisconsin (Diagnostic)

The data. The Breast Cancer Wisconsin (Diagnostic) dataset (Wolberg, Mangasarian, Street and Street, UCI Machine Learning Repository) has 569 rows, one per tumour sample, and 30 numeric features measured from a cell image. Each row is labelled benign or malignant. It ships with scikit-learn as load_breast_cancer, so anyone can re-run this.

The setup. We treat benign rows as "normal". The model is trained on two thirds of the benign rows only and never sees a malignant row. We then score the held-out benign rows (to see how often normal rows get flagged) and the malignant rows (to see how often the unusual class gets flagged). Because the labels exist, we can also compute the AUC: the chance that a randomly chosen malignant row gets a higher anomaly score than a randomly chosen held-out benign row (0.5 is a coin flip, 1.0 is a perfect ranking). We fit nine models: three values of gamma times three values of nu.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import OneClassSVM
from sklearn.metrics import roc_auc_score

X, y = load_breast_cancer(return_X_y=True)   # y: 0 = malignant, 1 = benign
benign, malignant = X[y == 1], X[y == 0]
print("benign:", len(benign), " malignant:", len(malignant), " features:", X.shape[1])

# Treat benign as "normal". Train on benign only; hold out a third of it.
train, held = train_test_split(benign, test_size=1/3, random_state=0)
scaler = StandardScaler().fit(train)          # fit the scaler on training rows only
Z_train, Z_held, Z_mal = (scaler.transform(a) for a in (train, held, malignant))
print("train benign:", len(train), " held-out benign:", len(held))
print("gamma='scale' here =", round(1 / (Z_train.shape[1] * Z_train.var()), 4))

print("gamma  nu    SVs    train<-0.001  benign-flagged  malignant-flagged  AUC")
for gamma in (0.01, "scale", 0.5):
    for nu in (0.01, 0.05, 0.20):
        m = OneClassSVM(kernel="rbf", gamma=gamma, nu=nu).fit(Z_train)
        sv = len(m.support_) / len(Z_train)
        tr = (m.decision_function(Z_train) < -0.001).mean()  # clearly outside
        hb = (m.predict(Z_held) == -1).mean()   # false-flag rate on new normal rows
        ml = (m.predict(Z_mal) == -1).mean()    # share of malignant rows flagged
        # anomaly score = minus the decision function, so higher = more unusual
        s = -np.concatenate([m.decision_function(Z_held), m.decision_function(Z_mal)])
        lab = np.r_[np.zeros(len(Z_held)), np.ones(len(Z_mal))]
        print("%-6s %.2f  %.3f  %.3f         %.3f           %.3f              %.3f"
              % (gamma, nu, sv, tr, hb, ml, roc_auc_score(lab, s)))

Output (scikit-learn 1.7.1, NumPy 2.3.2, Python 3.12):

benign: 357  malignant: 212  features: 30
train benign: 238  held-out benign: 119
gamma='scale' here = 0.0333
gamma  nu    SVs    train<-0.001  benign-flagged  malignant-flagged  AUC
0.01   0.01  0.071  0.000         0.067           0.892              0.968
0.01   0.05  0.084  0.021         0.067           0.887              0.969
0.01   0.20  0.206  0.189         0.185           0.948              0.966
scale  0.01  0.185  0.000         0.176           0.953              0.968
scale  0.05  0.185  0.000         0.176           0.953              0.968
scale  0.20  0.244  0.168         0.227           0.967              0.966
0.5    0.01  1.000  0.000         1.000           1.000              0.959
0.5    0.05  1.000  0.000         1.000           1.000              0.959
0.5    0.20  1.000  0.000         1.000           1.000              0.959

How to read the columns. SVs is the share of training rows that became support vectors. train<-0.001 is the share of training rows clearly outside the boundary (decision value below -0.001; rows on the boundary itself sit within rounding of zero, so we do not count them). benign-flagged is the false-flag rate on held-out normal rows. malignant-flagged is the share of the unusual class caught. AUC measures the ranking, independent of where the cut-off sits.

What the run shows.

• The nu bounds hold. In every row, the share of training rows clearly outside is at most nu, and the share of support vectors is at least nu.
• The flag rate follows the settings. With gamma = 0.01, moving nu from 0.05 to 0.20 raised the share of held-out benign rows flagged from 0.067 to 0.185, on the same data. With gamma = 0.5, every held-out benign row was flagged (1.000) at every nu: every training row became a support vector, so the boundary was drawn tightly around the training rows themselves and every held-out row fell outside it.
• The ranking barely moved. AUC stayed between 0.959 and 0.969 across all nine models. The settings mostly moved the cut-off, not the order of the rows.
• gamma="scale" is a formula. Here it works out to 0.0333, which is 1 / 30 features, because the standardized training data have a variance of about 1.

5. Key Points

6. Practical Guidance

6.1 When a one-class SVM fits

• You have a reasonably clean sample of normal behavior and few or no labeled anomalies.
• The number of training samples is moderate, so a kernel method is affordable.
• You can scale the features and inspect the flagged points.

6.2 When to look elsewhere

• The training data are heavily contaminated and cannot be cleaned: consider Isolation Forest or robust estimators.
• You have plenty of labeled examples of both classes: a supervised classifier will normally use that information better.
• The dataset is very large and you cannot subsample: consider a method with better scaling.
• You need a probability: the score is not one. A density model such as a Gaussian mixture gives a density, which is not the probability that a point is anomalous. Getting that probability needs labeled examples to calibrate against, or an explicit assumption about how common anomalies are.

6.3 Operating a model over time

Normal behavior changes. Monitor the share of new observations that are flagged and the distribution of scores, and refit on recent normal data when they drift. Keep a way for reviewers to mark flags as correct or not, so that you gain labels over time and can move to a supervised or semi-supervised method if enough accumulate.

7. Recommendations

These are general suggestions on how to apply the method. Apart from the point that cites Section 4.4, they are not drawn from experiments.

8. Conclusion

The one-class SVM is a kernel method that learns a region of feature space around normal examples and flags points outside it. Its behavior is governed by a small set of choices: the kernel, gamma and nu. Nu has a precise meaning as a pair of bounds, the kernel sets the shape of the region, and the decision function gives a ranking rather than a probability.

This paper is an explanation of the method. Its one run on real data, a single public dataset, shows that the flagged share follows nu and gamma, which you choose. Whether a one-class SVM works well on a given problem has to be checked on that problem, using held-out normal data, any available labels and expert review of what it flags. It also has real limits: it is sensitive to scaling and to contaminated training data, and it costs more as the data grow.

9. Limits of this paper

• The real-data example is one public dataset of 569 rows and 30 features, with one random split (random_state=0). Other splits and other data will give other numbers.
• In that dataset the "unusual" class is malignant tumours, which is not rare (212 of 569 rows). We used it because the labels let us check the ranking, not because it resembles a typical anomaly problem.
• We did not compare the one-class SVM with Isolation Forest, local outlier factor or other methods, so this page makes no claim about which works better.
• The example demonstrates a statistical method on public data. It is not medical advice and not a diagnostic test.
• Statements about computing cost come from the scikit-learn documentation, not from timings we ran.