Figure 1.1 Rules for the contact lens data.

Figure 1.2 Decision tree for the contact lens data.

Figure 1.3 Decision trees for the labor negotiations data. (a)(b)

Figure 2.1 A family tree and two ways of expressing the sister-of relation.

Figure 2.2 ARFF file for the weather data.

Figure 3.1 Decision tree for a simple disjunction.

Figure 3.2 The exclusive-or problem. If x=1 and y=0 then class = a If x=0 and y=1 then class = a If x=0 and y=0 then class = b If x=1 and y=1 then class = b

Figure 3.3 Decision tree with a replicated subtree. If x=1 and y=1 then class = a If z=1 and w=1 then class = a Otherwise class = b

Figure 3.4 Rules for the Iris data. Default: Iris-setosa 1 except if petal-length  2.45 and petal-length < and petal-width < then Iris-versicolor 4 except if petal-length  4.95 and petal-width < then Iris-virginica 6 else if sepal-length < 4.95 and sepal-width  then Iris-virginica 8 else if petal-length  then Iris-virginica10 except if petal-length < 4.85 and sepal-length < then Iris-versicolor12

Figure 3.5 The shapes problem. Shaded: standing Unshaded: lying

Figure 3.6(a) Models for the CPU performance data: linear regression. PRP = MYCT MMIN MMAX CACH CHMIN CHMAX

Figure 3.6(b) Models for the CPU performance data: regression tree.

Figure 3.6(c) Models for the CPU performance data: model tree.

Figure 3.7 Different ways of partitioning the instance space. (a)(b)(c)(d)

Figure 3.8 Different ways of representing clusters a b c d e f g h … (a)(b) (c)(d)

Figure 4.1 Pseudo-code for 1R.

Figure 4.2 Tree stumps for the weather data. (a)(b) (c)(d)

Figure 4.3 Expanded tree stumps for the weather data. (a)(b) (c)

Figure 4.4 Decision tree for the weather data.

Figure 4.5 Tree stump for the ID code attribute.

Figure 4.6 (a) Operation of a covering algorithm; (b) decision tree for the same problem. (a) (b)

Figure 4.7 The instance space during operation of a covering algorithm.

Figure 4.8 Pseudo-code for a basic rule learner.

Figure 5.1 A hypothetical lift chart.

Figure 5.2 A sample ROC curve.

Figure 5.3 ROC curves for two learning schemes.

Figure 6.1 Example of subtree raising, where node C is “raised” to subsume node B. (a)(b)

Figure 6.2 Pruning the labor negotiations decision tree.

Figure 6.3 Generating rules using a probability measure.

Figure 6.4 Definitions for deriving the probability measure. p number of instances of that class that the rule selects; t total number of instances that the rule selects; p total number of instances of that class in the dataset; t total number of instances in the dataset.

Figure 6.5 Algorithm for forming rules by incremental reduced error pruning.

Figure 6.6 Algorithm for expanding examples into a partial tree.

Figure 6.7 Example of building a partial tree. (a) (b) (c)

Figure 6.7 (continued) Example of building a partial tree. (d)(e)

Figure 6.8 Rules with exceptions, for the Iris data. Exceptions are represented as Dotted paths, alternatives as solid ones.

Figure 6.9 A maximum margin hyperplane.

Figure 6.10 A boundary between two rectangular classes.

Figure 6.11 Pseudo-code for model tree induction.

Figure 6.12 Model tree for a dataset with nominal attributes.

Figure 6.13 Clustering the weather data. (a) (c) (b)

Figure 6.13 (continued) Clustering the weather data. (d) (e)

Figure 6.13 (continued) Clustering the weather data. (f)

Figure 6.14 Hierarchical clusterings of the Iris data. (a)

Figure 6.14 (continued) Hierarchical clusterings of the Iris data. (b)

Figure 6.15 A two-calss mixture model. A 51 A 43 B 62 B 64 A 45 A 42 A 46 A 45 A 45 B 62 A 47 A 52 B 64 A 51 B 65 A 48 A 49 A 46 B 64 A 51 A 52 B 62 A 49 A 48 B 62 A 43 A 40 A 48 B 64 A 51 B 63 A 43 B 65 B 66 B 65 A 46 A 39 B 62 B 64 A 52 B 63 B 64 A 48 B 64 A 48 A 51 A 48 B 64 A 42 A 48 A 41 data model  A =50,  A =5, p A =0.6  B =65,  B =2, p B =0.4

Figure 7.1 Attribute space for the weather dataset.

Figure 7.2 Discretizing temperature using the entropy method.

Figure 7.3 The result of discretizing temperature no yes yes no yes yes yes no no yes yes no yes yes F E D C B A

Figure 7.4 Class distribution for a two-class, two-attribute problem.

Figure 7.5 Number of international phone calls from Belgium, 1950–1973.

Figure 7.6 Algorithm for bagging. model generation Let n be the number of instances in the training data. For each of t iterations: Sample n instances with replacement from training data. Apply the learning algorithm to the sample. Store the resulting model. classification For each of the t models: Predict class of instance using model. Return class that has been predicted most often.

Figure 7.7 Algorithm for boosting. model generation Assign equal weight to each training instance. For each of t iterations: Apply learning algorithm to weighted dataset and store resulting model. Compute error e of model on weighted dataset and store error. If e equal to zero, or e greater or equal to 0.5: Terminate model generation. For each instance in dataset: If instance classified correctly by model: Multiply weight of instance by e / (1 - e). Normalize weight of all instances. classification Assign weight of zero to all classes. For each of the t (or less) models: Add -log(e / (1 - e)) to weight of class predicted by model. Return class with highest weight.

Figure 8.1 Weather data: (a) in spreadsheet; (b) comma-separated. (a)(b)

Figure 8.1 Weather data: (c) in ARFF format. (c)

Figure 8.2 Output from the J4.8 decision tree learner.

Figure 8.3 Using Javadoc: (a) the front page; (b) the weka.core package. (a) (b)

Figure 8.4 A class of the weka.classifiers package.

Figure 8.5 Output from the M5´ program for numeric prediction.

Figure 8.6 Output from J4.8 with cost-sensitive classification.

Figure 8.7 Effect of AttributeFilter on the weather dataset.

Figure 8.8 Output from the APRIORI association rule learner.

Figure 8.9 Output from the EM clustering scheme.

Figure 8.10 Source code for the message classifier.

Figure 8.10 (continued)

Figure 8.11 Source code for the ID3 decision tree learner.

Figure 8.11 (continued)

Figure 8.11

Figure 8.12 Source code for a filter that replaces the missing values in a dataset.

Figure 8.12

Figure 9.1 Representation of Iris data: (a) one dimension.

Figure 9.1 Representation of Iris data: (b) two dimensions.

Figure 9.2 Visualization of classification tree for grasses data.