VISUALIZATION TECHNIQUES UTILIZING THE SENSITIVITY ANALYSIS OF MODELS Ivo Kondapaneni, Pavel Kordík, Pavel Slavík Department of Computer Science and Engineering,

Slides:

Advertisements

Similar presentations

Artificial Neural Networks

Advertisements

The Software Infrastructure for Electronic Commerce Databases and Data Mining Lecture 4: An Introduction To Data Mining (II) Johannes Gehrke

A Brief Overview of Neural Networks By Rohit Dua, Samuel A. Mulder, Steve E. Watkins, and Donald C. Wunsch.

Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques

Support Vector Machines

Groundwater 3D Geological Modeling: Solving as Classification Problem with Support Vector Machine A. Smirnoff, E. Boisvert, S. J.Paradis Earth Sciences.

Programming and Simulations Frank Witmer 6 January 2011.

Chapter 4: Linear Models for Classification

Institute of Intelligent Power Electronics – IPE Page1 Introduction to Basics of Genetic Algorithms Docent Xiao-Zhi Gao Department of Electrical Engineering.

x – independent variable (input)

RBF Neural Networks x x1 Examples inside circles 1 and 2 are of class +, examples outside both circles are of class – What NN does.

Data Mining Techniques Outline

Classification Dr Eamonn Keogh Computer Science & Engineering Department University of California - Riverside Riverside,CA Who.

Lazy Learning k-Nearest Neighbour Motivation: availability of large amounts of processing power improves our ability to tune k-NN classifiers.

Introduction to Neural Networks Simon Durrant Quantitative Methods December 15th.

Data mining and statistical learning - lecture 13 Separating hyperplane.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

General Mining Issues a.j.m.m. (ton) weijters Overfitting Noise and Overfitting Quality of mined models (some figures are based on the ML-introduction.

Results Comparison with existing approaches on benchmark datasets Evaluation on a uveal melanoma datasetEvaluation on the two-spiral dataset Evaluation.

M. Verleysen UCL 1 Feature Selection with Mutual Information and Resampling M. Verleysen Université catholique de Louvain (Belgium) Machine Learning Group.

Data Mining By Andrie Suherman. Agenda Introduction Major Elements Steps/ Processes Tools used for data mining Advantages and Disadvantages.

Data Mining Techniques

Attention Deficit Hyperactivity Disorder (ADHD) Student Classification Using Genetic Algorithm and Artificial Neural Network S. Yenaeng 1, S. Saelee 2.

A Simple Method to Extract Fuzzy Rules by Measure of Fuzziness Jieh-Ren Chang Nai-Jian Wang.

Chapter 13 Genetic Algorithms. 2 Data Mining Techniques So Far… Chapter 5 – Statistics Chapter 6 – Decision Trees Chapter 7 – Neural Networks Chapter.

COMP3503 Intro to Inductive Modeling

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

Evolving a Sigma-Pi Network as a Network Simulator by Justin Basilico.

Integrating Neural Network and Genetic Algorithm to Solve Function Approximation Combined with Optimization Problem Term presentation for CSC7333 Machine.

Cristian Urs and Ben Riveira. Introduction The article we chose focuses on improving the performance of Genetic Algorithms by: Use of predictive models.

Using Neural Networks in Database Mining Tino Jimenez CS157B MW 9-10:15 February 19, 2009.

Introduction to Data Mining Group Members: Karim C. El-Khazen Pascal Suria Lin Gui Philsou Lee Xiaoting Niu.

© Copyright 2004 ECE, UM-Rolla. All rights reserved A Brief Overview of Neural Networks By Rohit Dua, Samuel A. Mulder, Steve E. Watkins, and Donald C.

Introduction to machine learning and data mining 1 iCSC2014, Juan López González, University of Oviedo Introduction to machine learning Juan López González.

1 SUPPORT VECTOR MACHINES İsmail GÜNEŞ. 2 What is SVM? A new generation learning system. A new generation learning system. Based on recent advances in.

COMPARISON OF IMAGE ANALYSIS FOR THAI HANDWRITTEN CHARACTER RECOGNITION Olarik Surinta, chatklaw Jareanpon Department of Management Information System.

INTERACTIVE ANALYSIS OF COMPUTER CRIMES PRESENTED FOR CS-689 ON 10/12/2000 BY NAGAKALYANA ESKALA.

Chapter 6: Techniques for Predictive Modeling

Neural and Evolutionary Computing - Lecture 9 1 Evolutionary Neural Networks Design  Motivation  Evolutionary training  Evolutionary design of the architecture.

Using Interactive Evolution for Exploratory Data Analysis Tomáš Řehořek Czech Technical University in Prague.

KNN & Naïve Bayes Hongning Wang Today’s lecture Instance-based classifiers – k nearest neighbors – Non-parametric learning algorithm Model-based.

Alice E. Smith and Mehmet Gulsen Department of Industrial Engineering

Chapter 4 –Dimension Reduction Data Mining for Business Analytics Shmueli, Patel & Bruce.

Bump Hunting The objective PRIM algorithm Beam search References: Feelders, A.J. (2002). Rule induction by bump hunting. In J. Meij (Ed.), Dealing with.

FAKE GAME updates Pavel Kordík

KNN & Naïve Bayes Hongning Wang

An Evolutionary Algorithm for Neural Network Learning using Direct Encoding Paul Batchis Department of Computer Science Rutgers University.

CTU Prague, Faculty of Electrical Engineering, Department of Computer Science Pavel Kordík GMDH and FAKE GAME: Evolving ensembles of inductive models.

Inductive modelling: Detection of system states validity Pavel Kordík Department of Computer Science and Engineering, FEE,CTU Prague

GraphiCon 2008 | 1 Trajectory classification based on Hidden Markov Models Jozef Mlích and Petr Chmelař Brno University of Technology, Faculty of Information.

2011 Data Mining Industrial & Information Systems Engineering Pilsung Kang Industrial & Information Systems Engineering Seoul National University of Science.

The GAME Algorithm Applied to Complex Fractionated Atrial Electrograms Data Set Pavel Kordík, Václav Křemen and Lenka Lhotská Department of Computer Science.

Business Intelligence and Decision Support Systems (9 th Ed., Prentice Hall) Chapter 6: Artificial Neural Networks for Data Mining.

Inductive model evolved from data

Geographical Information Systems

OPTIMIZATION OF MODELS: LOOKING FOR THE BEST STRATEGY

School of Computer Science & Engineering

2006 IEEE World Congress on Computational Intelligence, International Joint Conference on Neural Networks (IJCNN) Evolutionary Search for Interesting Behavior.

ECON734: Spatial Econometrics – Lab 2

Regularization of Evolving Polynomial Models

CS621: Artificial Intelligence Lecture 17: Feedforward network (lecture 16 was on Adaptive Hypermedia: Debraj, Kekin and Raunak) Pushpak Bhattacharyya.

klinické neurofyziologie

ECON734: Spatial Econometrics – Lab 2

Department of Electrical Engineering

Data Mining, Machine Learning, Data Analysis, etc. scikit-learn

Introduction to Radial Basis Function Networks

Linear Discrimination

Primer on Neural networks

Presentation transcript:

VISUALIZATION TECHNIQUES UTILIZING THE SENSITIVITY ANALYSIS OF MODELS Ivo Kondapaneni, Pavel Kordík, Pavel Slavík Department of Computer Science and Engineering, Faculty of Eletrical Engineering, Czech Technical University in Prague, Czech Republic Presenting author: Pavel Kordík Ivo Kondapaneni, Pavel Kordík, Pavel Slavík Department of Computer Science and Engineering, Faculty of Eletrical Engineering, Czech Technical University in Prague, Czech Republic Presenting author: Pavel Kordík

2 Overview Motivation Data mining models Visualization based on sensitivity analysis Regression problems Classification problems Definition of interesting plots Genetic search for 2D and 3D plots

3 Motivation Data mining – extracting new, potentially useful information from data DM Models are automatically generated Are models always credible? Are models comprehensible? How to extract information from models? Visualization

4 Data mining models Often black-box models generated from data E.g. Neural networks What is inside? Data mining black box model Input variables Output variable (s)

5 Inductive model Estimates output from inputs Generated automatically Evolved by niching GA Grows from minimal form Contains hybrid units Several training methods Ensemble of models

6 Example: Housing data CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Per capita crime rate by town Proportion of owner-occupied units built prior to 1940 Weighted distances to five Boston employment centres Input variables Output variable Median value of owner-occupied homes in $1000's

7 Housing data – records CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Input variables Output variable … … …

8 Housing data – inductive model CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Input variables Output variable Niching genetic algorithm evolves units in first layer sigmoid MEDV=1/(1-exp(-5.724*CRIM )) sigmoid MEDV=1/(1-exp(-5.861*AGE )) Error: 0.13Error: 0.21

9 Housing data – inductive model CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Input variables Output variable Niching genetic algorithm evolves units in second layer sigmoid Error: 0.13Error: 0.21 sigmoid Error: 0.26 linear Error: 0.24 polyno mial MEDV=0.747*(1/(1-exp(-5.724*CRIM ))) *(1/(1-exp(-5.861*AGE ))) Error: 0.10

10 Housing data – inductive model CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Input variables Output variable sigmoid linear polyno mial polyno mial linear expo nential Error: 0.08 Constructed model has very low validation error!

11 Housing data – inductive model CRIM ZN INDUS NOX RM AGE DIS RAD TAX PTRATIO B LSTA MEDV Input variables Output variable S SSL P PL E Error: 0.08 MEDV=(exp((0.038* 3.451*(1/(1-exp(-5.724*CRIM )))*(1/(1- exp(2.413*DIS-2.581)))*(1/(1-exp(2.413*DIS-2.581)))+0.429*(1/(1- exp(-5.861*AGE )))+0.024*(1/(1-exp(2.413*DIS ))) *0.350*(1/(1-exp(-3.613*RAD-0.088))) *( 0.747*(1/(1-exp(-5.724*CRIM )))+0.582*(1/(1-exp( *AGE )))*(1/(1-exp(-5.861*AGE )))+0.016) *(1/(1-exp(-5.724*CRIM ))) *INDUS *LSTA )*0.860)*13.072) Math equation is not comprehensible any more – we have to treat it as a black box model!

12 Visualization based on sensitivity analysis GAME

13 Sensitivity analysis of inductive model of MEDV House no. 189House no. 164 What will happen with the value of house when criminality in the area decreases/increases? Credible output?

14 Ensemble of inductive models Random initialization Random initialization Developing on the same Developing on the same training set Training affect just well Training affect just well defined areas of input space Each model - unique architecture, Each model - unique architecture, similar complexity similar transfer functions similar transfer functions Similar behavior for well defined areas Similar behavior for well defined areas Different behavior – under-defined areas Different behavior – under-defined areas ykyk y k-1 y k+1 i = x 2 minmax GAME

15 Credibility of models: Artificial data set Credibility: the criterion is a dispersion of models` responses. Advantages: No need of the training data set, Modeling method success considered, Inputs importance considered.

16 Example: Models of hot water consumption

17 Cold water consumption, increasing humidity

18 Models on Housing data Single modelEnsemble of 10 models BeforeAfter

19 Classification problems Data: Setoza class Virginica class Versicolor class Petal length Petal width Blue GAME model (Iris Setoza class) output = 1 decision boundary output = 0

20 Credibility of classifiers

21 Overlapping models X ensemble

22 Random behavior filtered out BeforeAfter

23 Problem – how to find information in n-dim space? Multidimensional space of input variables What we are looking for? –Interesting relationship of IO variables –Regions of high sensitivity –Credible models (compromise response) Can we automate the search?

24 When a plot is interesting for us? x istart x isize xixi

25 Definition of interesting plot Minimal volume of the envelope p min Maximal sensitivity of the output to the change of x i input variable – y size max Maximal size of the area – x isize max

26 Multiobjective optimization Interestingness: Unknown variables: –x 1,x 2,..., x i-1,x i+1,…x n x istart, x isize We will use “Niching” genetic algorithm Chromosome: x 1 x 2... x i-1 x i+1 … x n x istart x isize

27 Niching GA on simple data GAME ensemble Training data Chromosome: x start x size fitness = x size * 1/p * y size fitness x size x start Very simple problem Search space is 2D, can be visualized

28 Niching GA locates also local optima Three subpopulations (niches) of individuals survived

29 Automated retrieval of plots showing interesting behavior Genetic Algorithm Genetic algorithm with special fitness function is used to adjust all other inputs (dimensions) Best so far individual found (generation 0 – 17)

30 Housing data – interesting plot retrieved BeforeAfter Low fitness High fitness

31 Conclusion Credible regression Credible classification Automated retrieval

32 Future work I Automated knowledge extraction from data

33 Future work II FAKE GAME framework

34 Future work III Just released as open source project –Automated data preprocessing –Automated model building, validation –Optimization methods –Visualization see and join us: