Machine Learning with WEKA

Copyright: Martin Kramer (mkramer@wxs.nl) WEKA: the bird Copyright: Martin Kramer (mkramer@wxs.nl) 2018/9/15 University of Waikato

WEKA: the software Machine learning/data mining software written in Java (distributed under the GNU Public License) Used for research, education, and applications Complements “Data Mining” by Witten & Frank Main features: Comprehensive set of data pre-processing tools, learning algorithms and evaluation methods Graphical user interfaces (incl. data visualization) Environment for comparing learning algorithms 2018/9/15 University of Waikato

WEKA: versions There are several versions of WEKA: WEKA 3.0: “book version” compatible with description in data mining book WEKA 3.2: “GUI version” adds graphical user interfaces (book version is command-line only) WEKA 3.3: “development version” with lots of improvements This talk is based on WEKA 3.4 WEKA 3.x 2018/9/15 University of Waikato

WEKA only deals with “flat” files @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... Flat file in ARFF format 2018/9/15 University of Waikato

WEKA only deals with “flat” files @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... numeric attribute nominal attribute class attribute 2018/9/15 University of Waikato

2018/9/15 University of Waikato 7

Explorer: pre-processing the data Data can be imported from a file in various formats: ARFF, CSV, C4.5, binary Data can also be read from a URL or from an SQL database (using JDBC) Pre-processing tools in WEKA are called “filters” WEKA contains filters for: Discretization, normalization, attribute selection , resampling, transforming and combining attributes, … 2018/9/15 University of Waikato

3 1 2 5 4 Find your data folder open iris.arff data set make sure file extension is .arff 2018/9/15 University of Waikato

The iris Dataset (鳶尾花資料集) Sources (Perhaps the best known dataset in pattern recognition) Creator: R.A. Fisher Donor: Michael Marshall Date: July, 1988 Number of Instances: 150 (50 in each of 3 classes) Number of Input Attributes: 4, numeric Missing Attribute Values: None Attribute Information sepal length (in cm) 花萼長度 sepal width (in cm) petal length (in cm) 花瓣長度 petal width (in cm) class: iris Setosa, iris Versicolour, iris Virginica (3 classes) 2018/9/15 10

鳶尾花經典分類 Iris Versicolour Iris Setosa Iris Virginica 2018/9/15 11

鳶尾花(iris, 愛麗絲) 科別：鳶尾科多年生草本 生長適溫約攝氏20~28度 特性： iris源於希臘語，意思是彩虹 有根莖鳶尾和球莖鳶尾之分 高30~40公分 葉劍形帶狀 三片花瓣，多為藍或紫色 三片花蕚，多為藍或紫色 《鳶尾花》是梵谷的知名作品之一 2018/9/15 12

鳶尾花 三片花瓣為白色，三片花萼為藍紫褐色 花莖外形與葉片相似 叢生，葉為細劍形 2018/9/15 13

2018/9/15 University of Waikato University of Waikato

Save as other file format as needed 2018/9/15 University of Waikato

2018/9/15 University of Waikato University of Waikato

2018/9/15 University of Waikato

Choose petallength 2018/9/15 University of Waikato

click 2018/9/15 University of Waikato University of Waikato

click 2018/9/15 University of Waikato University of Waikato

Click to change parameters 2018/9/15 University of Waikato University of Waikato

2018/9/15 University of Waikato University of Waikato

Click to read more description 2018/9/15 University of Waikato University of Waikato

2018/9/15 University of Waikato University of Waikato

2018/9/15 University of Waikato University of Waikato

2018/9/15 University of Waikato University of Waikato

Explorer: building “classifiers” Classifiers in WEKA are models to predict nominal or numeric quantities for a dependent variable Implemented learning schemes include: Decision trees, instance-based classifiers, support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, … 2018/9/15 University of Waikato

1 2 2018/9/15 University of Waikato

1 2 2018/9/15 University of Waikato

Click to change parameters 2018/9/15 University of Waikato

2018/9/15 University of Waikato

1 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Scroll to see other results 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Right click 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Right click 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Click to change Y coordinate Click to change Y coordinate Classification errors 2018/9/15 University of Waikato

2018/9/15 University of Waikato

1 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2 1 output neurons 2018/9/15 University of Waikato 51

1 2 hidden neurons & connection weights 2018/9/15 University of Waikato 52

2 1 Click to change parameters input neurons & connection weights 2018/9/15 University of Waikato 53

1 2 3 2018/9/15 University of Waikato

2018/9/15 University of Waikato 55

1 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

ROC curve 2018/9/15 University of Waikato

Use a numeric attribute as output 2018/9/15 University of Waikato

Choose petallength 2018/9/15 University of Waikato 64

2018/9/15 University of Waikato

1 2 Use M5P to predict petal length of a iris flower M5 Model trees and rules: Combines a decision tree with linear regression 2018/9/15 University of Waikato

3 1 2 When to use model 1, 2, 3 Model 1 Model 2 Model 3 Generate 3 models 2018/9/15 University of Waikato

Linear Model 1 Linear Model 2 Linear Model 3 2018/9/15 University of Waikato

right click “Visualize classifier error” performance evaluation right click “Visualize classifier error” 2018/9/15 University of Waikato

Click a data point to show the data window 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2 1 choose right click 2018/9/15 University of Waikato 72

2018/9/15 University of Waikato 73

Explorer: clustering data WEKA contains “clusterers” for finding groups of similar instances in a dataset Implemented schemes are: k-Means, EM, Cobweb, X-means, FarthestFirst Clusters can be visualized and compared to “true” clusters (if given) 2018/9/15 University of Waikato

1 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

3 1 2 2018/9/15 University of Waikato

Enter 3 clusters 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Clusters can be compared to “true” clusters (if given) 2018/9/15 University of Waikato

Scroll down 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Right click 2018/9/15 University of Waikato

2018/9/15 University of Waikato

Incorrectly classified instances Click to change Y coordinate Click to change Y coordinate Incorrectly classified instances 2018/9/15 University of Waikato

Explorer: finding associations WEKA contains an implementation of the Apriori algorithm for learning association rules Works only for discrete (categorical) data Can identify statistical dependencies between groups of attributes: milk, butter  bread, eggs (with confidence 0.9 and support 2000) Apriori can compute all rules that have a given minimum support and exceed a given confidence 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Load vote data set 2018/9/15 University of Waikato

The Congressional Voting Records Dataset (美國國會選舉資料集) Sources (Date: April, 1987) Origin : Congressional Quarterly Almanac, 98th Congress, 2nd session 1984 Donor: Jeff Schlimmer Area : Social Data Set Characteristics : Multivariate Associated Task : Classification Number of Instances: 435 Number of Input Attributes: 16, Categorical Number of classes : 2 (Republican or Democrat) Missing Attribute Values: Yes An unknown value is encoded as ? 2018/9/15 90

The soybean-large Dataset (黃豆資料集) Sources (Date: July, 1988) Origin : R.S. Michalski and R.L. Chilausky Donor: Ming Tan & Jeff Schlimmer Purpose : for soybean disease diagnosis Data Set Characteristics : Multivariate Associated Task : Classification Number of Instances: 307 Number of Input Attributes: 35, Categorical The values for attributes are encoded numerically Number of classes : 19 Only the first 15 of which have been used in prior work (The last 4 classes only have 17 records totally) Missing Attribute Values: Yes An unknown value is encoded as ? 2018/9/15 91

WEKA’s Association Mining Module Only Deals with Nominal Attribute @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... numeric attribute nominal attribute class attribute 2018/9/15 University of Waikato 92

WEKA’s Association Mining Module Can’t Be Started with Numeric Attributes @relation soybean-large-processed @attribute date {0,1,2,3,4,5,6} @attribute plant-stand {0,1} @attribute precip {0,1,2} @attribute temp {0,1,2} @attribute hail {0,1} @attribute crop-hist {0,1,2,3} @attribute area-damaged {0,1,2,3} @attribute severity {0,1,2} ……… @data 6,0,2,1,0,1,1,1,0,0,1,1,0,2,2,0,0,0,1,1,3,1,1,1,0,0,0,0,4,0,0,0,0,0,0,diaporthe-stem-canker 4,0,2,1,0,2,0,2,1,1,1,1,0,2,2,0,0,0,1,0,3,1,1,1,0,0,0,0,4,0,0,0,0,0,0,diaporthe-stem-canker nominal attributes (encoded numerically) 2018/9/15 University of Waikato 93

2018/9/15 University of Waikato

click 2018/9/15 University of Waikato

Expand the Window 2018/9/15 University of Waikato

2 minimum support = 0.45 minimum confidence = 0.9 Change parameters item support count rule confidence 2018/9/15 University of Waikato 97 Default : 10 rules

minimum support minimum confidence Set number of rules=15 2018/9/15 University of Waikato 98

click 15 rules 2018/9/15 University of Waikato

Explorer: attribute selection Panel that can be used to investigate which (subsets of) attributes are the most predictive ones Attribute selection methods contain two parts: A search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking An evaluation method: correlation-based, wrapper, information gain, chi-squared, … Very flexible: WEKA allows (almost) arbitrary combinations of these two 2018/9/15 University of Waikato 100

1 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Explorer: data visualization Visualization is very useful in practice, e.g. helps to determine difficulty of the learning problem WEKA can visualize single attributes (1D) and pairs of attributes (2D) Color-coded class values “Jitter” option to deal with nominal attributes (and to detect “hidden” data points) “Zoom-in” function 2018/9/15 University of Waikato

1 Load Glass data set 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2 1 Change PointSize 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2 1 Change PlotSize 2018/9/15 University of Waikato 113

Window size is changed when PlotSize is changed 2018/9/15 University of Waikato

Double click to enlarge 2018/9/15 University of Waikato 115

2018/9/15 University of Waikato

2018/9/15 University of Waikato

1 1 3 2 Choose Rectangle Zoom-in Click and drag to choose a bolck of data 2 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Performing experiments Experimenter makes it easy to compare the performance of different learning schemes For classification and regression problems Results can be written into file or database Evaluation options: cross-validation, learning curve, hold-out Can also iterate over different parameter settings Significance-testing is built in! 2018/9/15 University of Waikato

Click 2018/9/15 University of Waikato

Click 2018/9/15 University of Waikato

1. Set up parameters 2. Add a few data sets 3. Add a few algorithms 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Result message Running status 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

The Knowledge Flow GUI New graphical user interface for WEKA Java-Beans-based interface for setting up and running machine learning experiments Data sources, classifiers, etc. are beans and can be connected graphically Data “flows” through components: e.g., “data source” -> “filter” -> “classifier” -> “evaluator” Layouts can be saved and loaded again later 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Can continue this... 2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

2018/9/15 University of Waikato

Conclusion: try it yourself! WEKA is available at http://www.cs.waikato.ac.nz/ml/weka Also has a list of projects based on WEKA WEKA contributors: Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard Pfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger ,Ian H. Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg, Lucio de Souza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert , Richard Kirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy, Tony Voyle, Xin Xu, Yong Wang, Zhihai Wang 2018/9/15 University of Waikato