Presentation is loading. Please wait.

Presentation is loading. Please wait.

Anomaly detection with Bayesian networks Website: John Sandiford.

Published byBertina Mills Modified over 9 years ago

Similar presentations

Presentation on theme: "Anomaly detection with Bayesian networks Website: John Sandiford."— Presentation transcript:

1 Anomaly detection with Bayesian networks Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer John Sandiford

2 Contents Background What is anomaly detection? Bayesian networks Anomaly detection – supervised Anomaly detection – semi supervised Anomaly detection - time series Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

3 Background Mathematics Algorithms Data Mining Machine Learning Artificial Intelligence Bayesian networks – Research (Imperial College) – Software BAE Systems – Future concepts – Ground based diagnostics – Technical computing GE (General Electric) – Diagnostics – Prognostics – Reasoning New York Stock Exchange – Business Intelligence Bayes Server – Bayesian network software – Technical director Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

4 What is anomaly detection? System health monitoring – Advanced warning of mechanical failure Fault detection – Isolate faulty components Fraud detection – Can warn financial institutions of fraudulent transactions Pattern detection – Can detect unusual patterns Pre-processing – E.g. removal of unusual data, before building statistical models. Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer Anomaly detection, or outlier detection, is the process of identifying data which is unusual.

5 Types of anomaly detection Supervised – Labelled data – Specific faults Semi supervised – ‘Normal’ training data Unsupervised – Normal and abnormal training data Time Series can use all of the above Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

6 Bayesian networks Probabilistic Graphical Not a black box Handle conflicting evidence Unlike many rule based systems Multivariate Data driven and/or expert driven Missing data Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

7 Tasks & Models Tasks Classification Regression Clustering / Mixture models Density estimation Time series prediction Anomaly detection Decision Support Multivariate models Learning with missing data Probabilistic reasoning Models Multivariate Linear Regression Mixture models Time Series models – AR, Vector AR Hidden Markov Models Linear Kalman Filters Probabilistic PCA Factor Analysis Hybrid models – E.g. Mixtures of PPCA Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

8 Anomaly detection with Bayesian networks High dimensional data – Humans find difficult to interpret – Anomalies may not be visible on individual variables Discrete and continuous variables Allow missing data – Learning – Prediction/anomaly detection Temporal and non temporal variables in the same model Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

9 SUPERVISED ANOMALY DETECTION In this section we discuss anomaly detection with Bayesian networks, using labelled data Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

10 Comparison Advantages Learning is focused Prediction is specific and has an associated probability Diagnostics easier Disadvantages Anomalies tend to be different – Past anomalies may not predict future anomalies well Expense of labelling – E.g. Cost of experts Insufficient data labelled anomalous. It is too difficult to manually identify anomalous data. – Perhaps because the data is high dimensional, or is a complex time series or both. Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

11 Classification Multiple outputsMutually exclusive Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

12 Demonstration Identification network Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

13 Training Expert opinion Learn parameters from data Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

14 Prediction Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

15 Model performance & comparison Additional variables? BIC Confusion matrix Lift Chart Over fitting Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

16 Diagnostics? Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

17 SEMI SUPERVISED ANOMALY DETECTION In this section we discuss anomaly detection with Bayesian networks, using ‘normal’ training data. Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

18 Demonstration Mixture model Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

19 Prediction No data mapped to Cluster variable Missing data allowed Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

20 Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

21 Multivariate prediction (log-likelihood) Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

22 Multivariate prediction (conflict) Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

23 Time series Modelling time series data without a time series model Using a time series model Temporal & non temporal variables Classification, Regression, Log likelihood Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

24 Questions Website: www.BayesServer.com Email: john.sandiford@BayesServer.com Twitter: @BayesServer

Download ppt "Anomaly detection with Bayesian networks Website: John Sandiford."

Similar presentations

Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California

Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California
An Overview of Machine Learning

An Overview of Machine Learning
POSTER TEMPLATE BY: www.PosterPresentations.com Multi-Sensor Health Diagnosis Using Deep Belief Network Based State Classification Prasanna Tamilselvan.

POSTER TEMPLATE BY: Multi-Sensor Health Diagnosis Using Deep Belief Network Based State Classification Prasanna Tamilselvan.
What is Statistical Modeling

What is Statistical Modeling
Lecture 17: Supervised Learning Recap Machine Learning April 6, 2010.

Lecture 17: Supervised Learning Recap Machine Learning April 6, 2010.
CIS 678 Artificial Intelligence problems deduction, reasoning knowledge representation planning learning natural language processing motion and manipulation.

CIS 678 Artificial Intelligence problems deduction, reasoning knowledge representation planning learning natural language processing motion and manipulation.
A Framework for Discovering Anomalous Regimes in Multivariate Time-Series Data with Local Models Stephen Bay Stanford University, and Institute for the.

A Framework for Discovering Anomalous Regimes in Multivariate Time-Series Data with Local Models Stephen Bay Stanford University, and Institute for the.
Engineering Data Analysis & Modeling Practical Solutions to Practical Problems Dr. James McNames Biomedical Signal Processing Laboratory Electrical & Computer.

Engineering Data Analysis & Modeling Practical Solutions to Practical Problems Dr. James McNames Biomedical Signal Processing Laboratory Electrical & Computer.
Introduction. 1.Data Mining and Knowledge Discovery 2.Data Mining Methods 3.Supervised Learning 4.Unsupervised Learning 5.Other Learning Paradigms 6.Introduction.

Introduction. 1.Data Mining and Knowledge Discovery 2.Data Mining Methods 3.Supervised Learning 4.Unsupervised Learning 5.Other Learning Paradigms 6.Introduction.
Learning Programs Danielle and Joseph Bennett (and Lorelei) 4 December 2007.

Learning Programs Danielle and Joseph Bennett (and Lorelei) 4 December 2007.
Part I: Classification and Bayesian Learning

Part I: Classification and Bayesian Learning
Chapter 5 Data mining : A Closer Look.

Chapter 5 Data mining : A Closer Look.
Introduction to machine learning

Introduction to machine learning
Introduction to Data Mining Engineering Group in ACL.

Introduction to Data Mining Engineering Group in ACL.
CSC 478 Programming Data Mining Applications Course Summary Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

CSC 478 Programming Data Mining Applications Course Summary Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.
CS 60050 Machine Learning. What is Machine Learning? Adapt to / learn from data  To optimize a performance function Can be used to:  Extract knowledge.

CS Machine Learning. What is Machine Learning? Adapt to / learn from data  To optimize a performance function Can be used to:  Extract knowledge.
LLNL-PRES-671957 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

LLNL-PRES This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
General Information Course Id: COSC6342 Machine Learning Time: MO/WE 2:30-4p Instructor: Christoph F. Eick Classroom:SEC 201

General Information Course Id: COSC6342 Machine Learning Time: MO/WE 2:30-4p Instructor: Christoph F. Eick Classroom:SEC 201
Anomaly detection Problem motivation Machine Learning.

Anomaly detection Problem motivation Machine Learning.
Kansas State University Department of Computing and Information Sciences CIS 830: Advanced Topics in Artificial Intelligence From Data Mining To Knowledge.

Kansas State University Department of Computing and Information Sciences CIS 830: Advanced Topics in Artificial Intelligence From Data Mining To Knowledge.

Similar presentations

Ads by Google