Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Minqi Zhou © Tan,Steinbach, Kumar Introduction to Data Mining.

Published byTrevion Satchell Modified over 9 years ago

Similar presentations

Presentation on theme: "Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Minqi Zhou © Tan,Steinbach, Kumar Introduction to Data Mining."— Presentation transcript:

1 Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Minqi Zhou © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1

2 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2 Anomaly/Outlier Detection l What are anomalies/outliers? –The set of data points that are considerably different than the remainder of the data l Variants of Anomaly/Outlier Detection Problems –Given a database D, find all the data points x  D with anomaly scores greater than some threshold t –Given a database D, find all the data points x  D having the top- n largest anomaly scores f(x) –Given a database D, containing mostly normal (but unlabeled) data points, and a test point x, compute the anomaly score of x with respect to D l Applications: –Credit card fraud detection, telecommunication fraud detection, network intrusion detection, fault detection

3 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 3 Importance of Anomaly Detection Ozone Depletion History l In 1985 three researchers (Farman, Gardinar and Shanklin) were puzzled by data gathered by the British Antarctic Survey showing that ozone levels for Antarctica had dropped 10% below normal levels l Why did the Nimbus 7 satellite, which had instruments aboard for recording ozone levels, not record similarly low ozone concentrations? l The ozone concentrations recorded by the satellite were so low they were being treated as outliers by a computer program and discarded! Sources: http://exploringdata.cqu.edu.au/ozone.html http://www.epa.gov/ozone/science/hole/size.html

4 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 4 Anomaly Detection l Challenges –How many outliers are there in the data? –Method is unsupervised  Validation can be quite challenging (just like for clustering) –Finding needle in a haystack l Working assumption: –There are considerably more “normal” observations than “abnormal” observations (outliers/anomalies) in the data

5 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 5 Anomaly Detection Schemes l General Steps –Build a profile of the “normal” behavior  Profile can be patterns or summary statistics for the overall population –Use the “normal” profile to detect anomalies  Anomalies are observations whose characteristics differ significantly from the normal profile l Types of anomaly detection schemes –Graphical & Statistical-based –Distance-based –Model-based

6 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 6 Graphical Approaches l Boxplot (1-D), Scatter plot (2-D), Spin plot (3-D) l Limitations –Time consuming –Subjective

7 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 7 Convex Hull Method l Extreme points are assumed to be outliers l Use convex hull method to detect extreme values l What if the outlier occurs in the middle of the data?

8 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 8 Statistical Approaches l Assume a parametric model describing the distribution of the data (e.g., normal distribution) l Apply a statistical test that depends on –Data distribution –Parameter of distribution (e.g., mean, variance) –Number of expected outliers (confidence limit)

9 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 9 Grubbs’ Test l Detect outliers in univariate data l Assume data comes from normal distribution l Detects one outlier at a time, remove the outlier, and repeat –H 0 : There is no outlier in data –H A : There is at least one outlier l Grubbs’ test statistic: l Reject H 0 if:

10 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 10 Statistical-based – Likelihood Approach l Assume the data set D contains samples from a mixture of two probability distributions: –M (majority distribution) –A (anomalous distribution) l General Approach: –Initially, assume all the data points belong to M –Let L t (D) be the log likelihood of D at time t –For each point x t that belongs to M, move it to A  Let L t+1 (D) be the new log likelihood.  Compute the difference,  = L t (D) – L t+1 (D)  If  > c (some threshold), then x t is declared as an anomaly and moved permanently from M to A

11 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 11 Statistical-based – Likelihood Approach l Data distribution, D = (1 – ) M + A l M is a probability distribution estimated from data –Can be based on any modeling method (naïve Bayes, maximum entropy, etc) l A is initially assumed to be uniform distribution l Likelihood at time t:

12 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 12 Limitations of Statistical Approaches l Most of the tests are for a single attribute l In many cases, data distribution may not be known l For high dimensional data, it may be difficult to estimate the true distribution

13 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 13 Distance-based Approaches l Data is represented as a vector of features l Three major approaches –Nearest-neighbor based –Density based –Clustering based

14 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 14 Nearest-Neighbor Based Approach l Approach: –Compute the distance between every pair of data points –There are various ways to define outliers:  Data points for which there are fewer than p neighboring points within a distance D  The top n data points whose distance to the kth nearest neighbor is greatest  The top n data points whose average distance to the k nearest neighbors is greatest

15 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 15 Outliers in Lower Dimensional Projection l In high-dimensional space, data is sparse and notion of proximity becomes meaningless –Every point is an almost equally good outlier from the perspective of proximity-based definitions l Lower-dimensional projection methods –A point is an outlier if in some lower dimensional projection, it is present in a local region of abnormally low density

16 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 16 Outliers in Lower Dimensional Projection l Divide each attribute into  equal-depth intervals –Each interval contains a fraction f = 1/  of the records l Consider a k-dimensional cube created by picking grid ranges from k different dimensions –If attributes are independent, we expect region to contain a fraction f k of the records –If there are N points, we can measure sparsity of a cube D as: –Negative sparsity indicates cube contains smaller number of points than expected

17 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 17 Example l N=100,  = 5, f = 1/5 = 0.2, N  f 2 = 4

18 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 18 Density-based: LOF approach l For each point, compute the density of its local neighborhood l Compute local outlier factor (LOF) of a sample p as the average of the ratios of the density of sample p and the density of its nearest neighbors l Outliers are points with largest LOF value p 2  p 1  In the NN approach, p 2 is not considered as outlier, while LOF approach find both p 1 and p 2 as outliers

19 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 19 Clustering-Based l Basic idea: –Cluster the data into groups of different density –Choose points in small cluster as candidate outliers –Compute the distance between candidate points and non-candidate clusters.  If candidate points are far from all other non-candidate points, they are outliers

20 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 20 Base Rate Fallacy l Bayes theorem: l More generally:

21 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 21 Base Rate Fallacy (Axelsson, 1999)

22 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 22 Base Rate Fallacy l Even though the test is 99% certain, your chance of having the disease is 1/100, because the population of healthy people is much larger than sick people

23 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 23 Base Rate Fallacy in Intrusion Detection l I: intrusive behavior,  I: non-intrusive behavior A: alarm  A: no alarm l Detection rate (true positive rate): P(A|I) l False alarm rate: P(A|  I) l Goal is to maximize both –Bayesian detection rate, P(I|A) –P(  I|  A)

24 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 24 Detection Rate vs False Alarm Rate l Suppose: l Then: l False alarm rate becomes more dominant if P(I) is very low

25 © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 25 Detection Rate vs False Alarm Rate l Axelsson: We need a very low false alarm rate to achieve a reasonable Bayesian detection rate

Download ppt "Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Minqi Zhou © Tan,Steinbach, Kumar Introduction to Data Mining."

Similar presentations

Brief introduction on Logistic Regression

Brief introduction on Logistic Regression
ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.

ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.
Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.

Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.
Data Mining Anomaly Detection

Data Mining Anomaly Detection
Likelihood Ratio, Wald, and Lagrange Multiplier (Score) Tests

Likelihood Ratio, Wald, and Lagrange Multiplier (Score) Tests
Mining Distance-Based Outliers in Near Linear Time with Randomization and a Simple Pruning Rule Stephen D. Bay 1 and Mark Schwabacher 2 1 Institute for.

Mining Distance-Based Outliers in Near Linear Time with Randomization and a Simple Pruning Rule Stephen D. Bay 1 and Mark Schwabacher 2 1 Institute for.
Data Mining: Concepts and Techniques (3rd ed.) — Chapter 12 —

Data Mining: Concepts and Techniques (3rd ed.) — Chapter 12 —
Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.

Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.
Anomaly Detection brief review of my prospectus Ziba Rostamian CS590 – Winter 2008.

Anomaly Detection brief review of my prospectus Ziba Rostamian CS590 – Winter 2008.
Anomaly Detection. Anomaly/Outlier Detection  What are anomalies/outliers? The set of data points that are considerably different than the remainder.

Anomaly Detection. Anomaly/Outlier Detection  What are anomalies/outliers? The set of data points that are considerably different than the remainder.
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by.
Visual Recognition Tutorial

Visual Recognition Tutorial
Independent Sample T-test Often used with experimental designs N subjects are randomly assigned to two groups (Control * Treatment). After treatment, the.

Independent Sample T-test Often used with experimental designs N subjects are randomly assigned to two groups (Control * Treatment). After treatment, the.
Inferences About Process Quality

Inferences About Process Quality
MACHINE LEARNING 6. Multivariate Methods 1. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Motivating Example  Loan.

MACHINE LEARNING 6. Multivariate Methods 1. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Motivating Example  Loan.
Chapter 9 Hypothesis Testing II. Chapter Outline  Introduction  Hypothesis Testing with Sample Means (Large Samples)  Hypothesis Testing with Sample.

Chapter 9 Hypothesis Testing II. Chapter Outline  Introduction  Hypothesis Testing with Sample Means (Large Samples)  Hypothesis Testing with Sample.
Sampling Distributions & Point Estimation. Questions What is a sampling distribution? What is the standard error? What is the principle of maximum likelihood?

Sampling Distributions & Point Estimation. Questions What is a sampling distribution? What is the standard error? What is the principle of maximum likelihood?
The Neymann-Pearson Lemma Suppose that the data x 1, …, x n has joint density function f(x 1, …, x n ;  ) where  is either  1 or  2. Let g(x 1, …,

The Neymann-Pearson Lemma Suppose that the data x 1, …, x n has joint density function f(x 1, …, x n ;  ) where  is either  1 or  2. Let g(x 1, …,
Jeff Howbert Introduction to Machine Learning Winter 2014 1 Anomaly Detection Some slides taken or adapted from: “Anomaly Detection: A Tutorial” Arindam.

Jeff Howbert Introduction to Machine Learning Winter Anomaly Detection Some slides taken or adapted from: “Anomaly Detection: A Tutorial” Arindam.
Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.

Data Mining Anomaly Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to.

Similar presentations

Ads by Google