Anomaly Detection: Identifying Rare and Unusual Patterns in Data

👁️‍🗨️ Anomaly Detection

Machine learning technique used to identify unusual data points that do not resemble the majority of the data.

Key Idea

We model normal behavior using Gaussian density estimation.
Any example with very low probability under this model is considered anomalous.

The goal is to detect rare or abnormal events.

It is typically treated as an unsupervised learning problem because:

• You usually have many examples of normal data.
• You rarely have labeled examples of anomalies.

Core Idea

Anomaly detection identifies unusual data points that do not resemble most of the data.

The idea is simple:

• Learn what “normal” looks like.
• Flag anything that looks very unlikely.

The model learns:

• High probability regions → where most normal data lies
• Low probability regions → unusual areas

Anomalies naturally fall into low-density regions.

---

Anomaly Detection Algorithm

Even though anomaly detection is mostly unsupervised,
a small labeled dataset is extremely useful for evaluation and tuning.

1. 📚 Train a Probability Model

Given unlabeled data:

$$x^{(1)}, x^{(2)}, ..., x^{(m)}$$

Split the data into

1. Training set 🦾

Used to learn the probability model $p(x)$.

Model the probability distribution of the data:

$$p(x)$$

Find a small number $\varepsilon$ such that:

• If $p(x) < \varepsilon$ → Anomaly
• Else → Normal

$\varepsilon$ is the threshold for flagging anomalies.

Normal data should have high probability and fit inside distribution, while anomalies should have low probability and come outside distribution.

2. Cross-validation 📋

Used to tune $\varepsilon$ select features

• Fine Tune using Cross-validation set

3. Test sets 🧪

Used for final evaluation only.

• Evaluate the final model on the test set to estimate real-world performance.
• Do not use the test set for tuning or model selection to avoid overfitting.

---

2. 🔎 Evaluate New Data $(x_{test})$

Flag a new example $x_{test}$ as an anomaly if:

Decision rule:

• If $p(x_{test}) < \varepsilon$ → Anomaly
• Else → Normal

Where:

• $p(x)$= probability of the example under the learned model
• $\varepsilon$ = small threshold value

If the probability is very low, the example is considered unusual.

---

Common Applications

1. Fraud Detection

Used to detect:

• Unusual login behavior
• Suspicious transactions
• Compromised accounts

Possible features:

• Number of logins
• Number of transactions
• Pages visited
• Typing speed
• User activity patterns

2. Manufacturing

Used to detect defective products such as:

• Aircraft engines
• Industrial components
• Hardware parts

3. Data Center Monitoring

Used to monitor server behavior:

Features may include:

• CPU usage
• Memory usage
• Disk activity
• Network traffic
• Derived metrics

---

Anomaly Detection vs Supervised Learning

Aspect	Supervised Learning	Anomaly Detection
Main Goal	Predict known classes	Detect unusual/rare examples
Training Data	Labeled data	Mostly normal data
Labels Required	Yes	Usually no anomaly labels needed
Learns	Decision boundary between classes	What “normal” behavior looks like
Mathematical Idea	Learn (P(y \mid x))	Learn (P(x))
Positive Examples	Need many examples	Often very few or none
Works Best When	Classes are well-defined	Anomalies are rare/unpredictable
Typical Dataset	Balanced or moderately imbalanced	Highly imbalanced
Output	Class label	Anomaly score / probability
Example Output	Spam / Not Spam	Normal / Suspicious
Handles New Unknown Attacks?	Usually poorly	Better
Common Algorithms	Logistic Regression, SVM, Neural Networks	Gaussian Models, Isolation Forest, One-Class SVM
Example Use Cases	Spam detection, image classification	Fraud detection, server monitoring
Fraud Detection Suitability	Good if many fraud examples exist	Better when fraud patterns constantly change
Manufacturing Defects	Less ideal with few defect examples	Very effective
Cybersecurity Intrusions	Hard when attacks evolve	Commonly used
Data Requirement	Large labeled datasets	Mostly normal operational data
Decision Process	Compare classes	Detect deviations from normal
Typical Assumption	All classes are represented in training	Normal behavior dominates data
Real-World Analogy	“Is this cat or dog?”	“This looks strange.”

---

What Comes Next

Typically, anomaly detection uses:

• The Gaussian (Normal) distribution
• Probability modeling techniques
• Threshold selection methods

Next steps usually involve:

1. Modeling each feature using a Gaussian distribution
2. Combining them into a joint probability model
3. Using that model to compute ( p(x) )

---

Related Posts

• Gaussian Distribution — the probability model used to score each feature; understanding the univariate Gaussian is the prerequisite for anomaly detection
• Multivariate Gaussian Distribution — captures correlations between features for better anomaly detection when features are not independent
• K-Means Clustering — the clustering approach anomaly detection is often contrasted with