Lecture 08 Classification-based Learning Topics Basics Decision Trees Multi-Layered Perceptrons Applications

Basics Machine Learning Classification Inductive learning Deductive learning Abductive learning (reasoning) Reinforcement learning Collaborative learning Classification Given a set of examples, each labelled as in a specific class, learn how we classify the examples Supervised learning

Basics Decision Trees Multi-Layered Perceptrons (MLP) A symbolic representation of of a reasoning process. It describes a data set by a tree-like structure. Multi-Layered Perceptrons (MLP) A subsymbolic representation architecture with multiple layers of preceptrons Parallel inference Learning by generalizing patterns to approximate functions

Decision Trees Example

Decision Trees Data representation: Attribute-based language Attribute-list and two examples: Alist: [Gender Age Blood Smoking Caffeine HT?] Ex1: [Male 50-59 high ≥1pack ≥3cups high] Ex2: [Female 50-59 low ≥1pack ≥3cups normal] Where Blood: Blood-pressure Caffeine: Caffeine-intake HT: Hypertension, class label

Decision Trees Knowledge representation: Tree-like structure The tree always starts from the root node and grows down by splitting the data at each level into new nodes according to some predictor (attribute, feature). The root node contains the entire data set (all data records), and child nodes hold respective subsets of that set. A split in a decision tree corresponds to the predictor with the maximum separating power. The best split does the best job in creating nodes where a single class dominates. Two of the best known methods of calculating the predictor’s power: Gini coefficient Entropy

Decision Trees The Gini coefficient is calculated as the area between the Lorenz curve and the diagonal divided by the area below the diagonal, i.e., (A-B)/B. The Gini coefficient ranges from 0 (perfect equality) to 1 (perfect inequality). Looking for largest Gini coefficient Brown formula:

Decision Trees Selecting an optimal tree with Gini splitting

Decision Trees Calculation of Gini coefficient for Class A (with 7 leaf nodes forming a Lorenz curve) (Non-increasingly) sorted instances in Class A 59, 23, 11, 4, 2, 1, 0 Corresponding cumulative percentages for Class A 59/100, 82/100, 93/100, 97/100, 99/100, 100/100, 100/100 Corresponding cumulative percentages for total population 60/150, 86/150, 97/150, 102/150, 105/150, 114/150, 150/150 G(A) = |1- [60/150 * 59/100 + (86-60)/150 * (59+82)/ 100 + (97-86)/150 * (82+93)/ 100 + (102-97)/150 * (97+93)/ 100 + (105-102)/150 * (99+97)/ 100 + (114-105)/150 * (100+99)/ 100 + (150-114)/150 * (100+ 100)/ 100]| =0.311

Decision Trees Gain chart of Class A

Decision Trees Selecting an optimal tree with random splitting

Decision Trees Extracting rules from decision trees The path from the root node to a bottom leaf reveals a decision rule For example, a rule associated with the right bottom leaf in the figure that represents Gini splits can be represented as follows: if (Predictor 1 = no) and (Predictor 4 = no) and (Predictor 6 = no) then class = Class A

Decision Trees Entropy Node A contains n classes, ci, i = 1, …, n, each with probability p(ci) - Entropy of node A: - Entropy of child nodes of A by some split (CNi: child node i; p(CNi): probability of CNi): - Gain of the split at node A:

Decision Trees Select the split with largest gain. When to end splitting? Calculate deviation D of the split against random splitting - p’ and n’: expected instances in class p and n if randomly distributed Null hypothesis: if random splitting, the deviation D will be distributed according to distribution with n-1 degrees of freedom

Decision Trees distribution (pdf) Stop splitting if D(n) ≤ r: degree of freedom χ2 p value r 0.25 0.20 0.15 0.10 0.05 0.025 0.02 0.01 0.005 0.0025 0.001 0.0005 1 1.32 1.64 2.07 2.71 3.84 5.02 5.41 6.63 7.88 9.14 10.83 12.12 2 2.77 3.22 3.79 4.61 5.99 7.38 7.82 9.21 10.60 11.98 13.82 15.20 3 4.11 4.64 5.32 6.25 7.81 9.35 9.84 11.34 12.84 14.32 16.27 17.73 Stop splitting if D(n) ≤

Decision Trees The main advantage of the decision-tree approach to classification is it visualises the solution; it is easy to follow any path through the tree. Relationships learned by a decision tree can be expressed as a set of rules, which can then be used in developing an intelligent system.

Decision Trees Data preprocessing Continuous data, such as age or income, have to be grouped into ranges, which can unwittingly hide important patterns. Missing or inconsistent data have to be brought back or resolved. Inability to examine more than one variable at a time. This confines trees to only the problems that can be solved by dividing the solution space into several successive rectangles

Multi-Layered Perceptrons Example

Multi-Layered Perceptrons Emulating biological neural network

Multi-Layered Perceptrons Data representation: Coded attribute-based language Each input node takes an attribute (feature) Coded attribute-list and two examples: A-list: [Gender Age Blood Smoking Caffeine HT?] Ex1: [0 0.5 1 1 1 1] Ex2: [1 0.5 0 1 1 0] Gender (categorical data): 0: male; 1:female Age (continuous data ): age/100 (or 0.1:<10 yrs; 0.2: 20~21; …; 0.9: 90~99; 1.0: >99) Blood (Blood-pressure): 0: low; 1: high Smoking: cigarettes /40 (or 0 for 0 cigarettes; 0.1: <10cigarretes; 0.5: 10~19; 1: ≥ 1 pack) Caffeine (Caffeine-intake continuous data): cups/3 (or 0: 0 cups; 0.5: 1~2 cups; 1.0: ≥ 3cups ) HT (Hypertension, class label): 0: normal; 1: high

Multi-Layered Perceptrons Knowledge representation: weight, neuron, and network structure

Multi-Layered Perceptrons Neuron structure: A neuron computes the weighted sum of the input signals and compares the result with a threshold value, . If the net input is less than the threshold, the neuron output is –1. But if the net input is greater than or equal to the threshold, the neuron becomes activated and its output attains a value +1. The neuron uses the following transfer or activation function: Y(X) is called a sign function, Ysign.

Multi-Layered Perceptrons Sample activation functions

MLP - Perceptrons Network structure: Preceptrons A perceptron is the simplest form of a neural network, consisting of a single neuron with adjustable synaptic weights and a hard limiter A single-layer two input perceptron

MLP - Perceptrons The aim of the perceptron is to classify inputs, x1, x2, . . ., xn, into one of two classes, say Y=A1 or Y=A2. In the case of an elementary perceptron, the n-dimensional input space is divided by a hyperplane into two decision regions. The hyperplane is defined by the linearly separable function:

MLP - Perceptrons cf: ax+by=c cf: ax+by+cz=d

MLP - Perceptrons A perceptron learn its classification by making small adjustments in the weights to reduce the difference (error) between the desired and actual outputs of the perceptron. The initial weights are randomly assigned, usually in a small range, and then updated to obtain the output consistent with the training examples. If the error is positive, we need to increase perceptron output; if it is negative, we need to decrease perceptron output.

MLP - Perceptron learning algorithm Step 1: Initialization Set initial weights w1, w2,…, wn and threshold  to random numbers in the range [0.5, 0.5].

MLP - Perceptron learning algorithm Step 2: Activation (a) Activate the perceptron by applying inputs x1(p), x2(p),…, xn(p) and desired output Yd (p). Calculate the actual output at iteration p = 1: where n is the number of the perceptron inputs. (b) Calculate the output error:

MLP - Perceptron learning algorithm Step 3: Weight training Calculate the weight correction at iteration p, Dwi(p), using delta rule: Update the weights of the perceptron Step 4: Iteration Increase iteration p by one, go back to Step 2 and repeat the process until convergence Epoch: An epoch finishes the weight adjustment with the whole training examples.

Multi-Layered Perceptrons Linearly inseparable problem with perceptrons Increasing layers: from perceptrons to MLP An MLP is a feedforward neural network with one or more hidden layers The network consists of an input layer of source neurons, at least one middle or hidden layer of computational neurons, and an output layer of computational neurons. The input signals are propagated in a forward direction on a layer-by-layer basis.

Multi-Layered Perceptrons MLP with two hidden layers

Multi-Layered Perceptrons Learning in an MLP proceeds the same way as for a perceptron. First, a training input pattern is presented to the network input layer. The network propagates the input pattern from layer to layer until the output pattern is generated by the output layer. If this pattern is different from the desired output, an error is calculated and then propagated backwards through the network from the output layer to the input layer. The weights are modified as the error is propagated - Back-propagation learning algorithm (BP)

Multi-Layered Perceptrons

MLP - BP Learning Algorithm Step 1: Initialization Set all the weights and threshold levels of the network to random numbers uniformly distributed inside a small range: where Fi is the total number of inputs of neuron i in the network. The weight initialization is done on a neuron-by-neuron basis.

MLP - BP Learning Algorithm Step 2: Activation Activate MLP by applying inputs x1(p), x2(p),…, xn(p) and desired outputs yd,1(p), yd,2(p),…, yd,l(p). (a) Calculate the actual outputs of the neurons in the hidden layer: where j = 1, .., m.

MLP - BP Learning Algorithm Step 2: Activation (continued) (b) Calculate the actual outputs of the neurons in the output layer: where k = 1, …, l. (c) Calculate the output errors of the neurons in the output layer:

MLP - BP Learning Algorithm Step 3: Weight training (a) Calculate the error gradient for the neurons in the output layer: Calculate the weight corrections by delta rule: Update the weights at the output neurons:

MLP - BP Learning Algorithm Step 3: Weight training (continued) (b) Calculate the propagated errors for the neurons in the hidden layer: Calculate the error gradient for the neurons in the hidden layer: Calculate the weight corrections by delta rule: Update the weights at the hidden neurons:

MLP - BP Learning Algorithm Step 4: Iteration Increase iteration p by one, go back to Step 2 and repeat the process until the selected error criterion is satisfied.

Multi-Layered Perceptrons We can accelerate training by including a momentum term in the delta rule and changing it into the generalized delta rule: where  is a positive number (0    1), called the momentum constant. Typically, the momentum constant is set to 0.95.

Multi-Layered Perceptrons To accelerate the convergence and yet avoid the danger of instability, we can apply two heuristics: Heuristic 1 If the change of the sum of squared errors has the same algebraic sign for several consequent epochs, then the learning rate parameter, , should be increased. The sum of squared errors (network performance measure): Heuristic 2 If the algebraic sign of the change of the sum of squared errors alternates for several consequent epochs, then the learning rate parameter, , should be decreased.

Multi-Layered Perceptrons Adapting the learning rate If the sum of squared errors at the current epoch exceeds the previous value by more than a predefined ratio (typically 1.04), the learning rate parameter is decreased (typically by multiplying by 0.7) and new weights and thresholds are calculated. If the error is less than the previous one, the learning rate is increased (typically by multiplying by 1.05).

Applications Decision trees Multi-Layered Perceptrons Data mining Churn model construction Multi-Layered Perceptrons Function approximation Pattern recognition Hand-writing recognition Case-based retrieval