1. CSE573 Autumn 1997 1 03/09/98 Machine Learning Administrative –Last topic: Decision Tree Learning Reading: 5.1, 5.4 Last time –finished NLP sample system’s “semantics” hasty conclusion This time –machine learning in general –decision-tree learning in particular

2. CSE573 Autumn 1997 2 Machine Learning in General Most general statement of the problem(s) –a computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experiences with E World Model (Predictor) Problem Solver (Control Rules) External World Concept learning Reinforcement learning Explanation-Based learning

3. CSE573 Autumn 1997 3 Components of a Learning System The “function” to be learned –a logical expression describing a concept (concept- or inductive learning):  x. chair(x)  has-back(x)  can-support(x)  … –a “policy” mapping from percepts to behaviors or actions (reinforcement learning) –search control rules or ranking function (explanation-based learning) The inputs (training data) Feedback –whether or not chair(OBJ-25) –state description resulting from executing an action –reward/penalty for executing a plan –results of a problem-solving episode (quality of solution, time to solution) Restriction on the form of the function (bias)

4. CSE573 Autumn 1997 4 Learning as Function Induction Given a set of tuples (the training set) –(x 11, x 12, …, x 1n, y 1 ) –(x 21, x 22, …, x 2n, y 2 ) –… Produce a function f(x 1, …, x n ) = y –a compact function –that works on unobserved examples, not just the training set Without some restrictions on f’s functional form, this will be impossible –numeric functions: linear, polynomial –symbolic functions: disjunctive normal form Inevitable tradeoff between the complexity of the function learned and the computational complexity of learning it.

5. CSE573 Autumn 1997 5 Decision-Tree Learning The function learned is disjunctive-normal-form formulas –(a 1 = v 11 ) ^ (a 2 = v 12 ) ^ … ^ (a n = v 1n ) => y = y 1k –(a 1 = v 21 ) ^ (a 2 = v 22 ) ^ … ^ (a n = v 2n ) => y = y 2k (often the v ij and y ij values are either TRUE or FALSE) The tree representation has an attribute at each node and one child for each value that attribute can take on. A BCT DEEF TFFF A=a 1 A=a 2 A=a 3 B=T B=F

6. CSE573 Autumn 1997 6 Example

7. CSE573 Autumn 1997 7 Basic Algorithm Recall, a node in the tree represents a conjunction of attribute values. We will try to build “the shortest possible” tree that classifies all the training examples correctly. In the algorithm we also store the list of attributes we have not used so far for classification. Initialization: tree  {} attributes  {all attributes} examples  {all training examples} Recursion: –Choose a new attribute A with possible values {a i } –For each a i, add a subtree formed by recursively building the tree with the current node as root all attributes except A all examples where A=a i

8. CSE573 Autumn 1997 8 Basic Algorithm (cont.) Termination (working on a single node): –If all examples have the same classification, then this combination of attribute values is sufficient to classify all (training) examples. Return the unanimous classification. –If examples is empty, then there are no examples with this combination of attribute values. Associate some “guess” with this combination. –If attributes is empty, then the training data is not sufficient to discriminate. Return some “guess” based on the remaining examples.

9. CSE573 Autumn 1997 9 Some Sample Attribute Choices D1,D2,...,D14 D1D2D14... DAY D1,D2,...,D14 ALL D1,D2,...,D14 HUMIDITY D1, S2, D3, D4, D8, D14 D5, D6, D7 D9, D10, D11 D12, D13 = high = normal D1,D2,...,D14 OUTLOOK D1, D3, D8, D9, D11 D3, D7, D12, D13 = sunny= overcast= rain D4, D5, D6, D10, D14

10. CSE573 Autumn 1997 10 Remaining questions Special cases –When attributes are left but no example (bad coverage in the training set) –When examples are left but no attributes (bad set of predictive attributes) How to choose the next attribute to discriminate on?

11. CSE573 Autumn 1997 11 How to choose the next attribute What is our goal in building the tree in the first place? –Maximize accuracy over the entire data set –Minimize expected number of tests to classify an example (In both cases this can argue for building the shortest tree.) We can’t really do the first looking only at the training set: we can only build a tree accurate for our subset and assume the characteristics of the full data set are the same. To minimize the expected number of tests –the best test would be one where each branch has all positive or all negative instances –the worst test would be one where the proportion of positive to negative instances is the same in every branch knowledge of A would provide no information about the example’s ultimate classification

12. CSE573 Autumn 1997 12 The Entropy (Disorder) of a Collection Suppose S is a collection containing positive and negative examples of the target concept: –Entropy(S)  – (p + log 2 p + + p - log 2 p - ) –where p + is the fraction of examples that are positive and p - is the fraction of examples that are negative Good features –minimum of 0 where p + = 0 and where p - = 0 –maximum where p + = p - = 0.5 Interpretation: the minimum number of bits required to encode the classification of an arbitrary member of S. We want to reduce the entropy in the collection as quickly as possible.

13. CSE573 Autumn 1997 13 Entropy and Information Gain The best attribute is one that maximizes the expected decrease in entropy –if entropy decreases to 0, the tree need not be expanded further –if entropy does not decrease at all, the attribute was useless Gain is defined to be –Gain(S, A) = Entropy(S) –  v  values(A) p {A=v} Entropy(S {A=v} ) –where p {A=v} is the proportion of S where A=v, and –S {A=v} is the collection taken by selecting those elements of S where A=v

14. CSE573 Autumn 1997 14 Example S: [9+, 5-] E = 0.940

15. CSE573 Autumn 1997 15 Choosing the First Attribute Humidity S: [9+, 5-] E = 0.940 HighLow S: [3+, 4-] E = 0.985 S: [6+, 1-] E = 0.592 Wind S: [9+, 5-] E = 0.940 HighLow S: [6+, 2-] E = 0.811 S: [3+, 3-] E = 1.000 Gain(S, Humidity) =.940 - (7/14).985 - (7/14).592 =.151 Gain(S, Wind) =.940 - (8/14).811 - (6/14)1.00 =.048 Gain(S, Outlook) =.246 Gain(S, Temperature) =.029

16. CSE573 Autumn 1997 16 After the First Iteration Outlook SunnyRain ? Yes ? Overcast D1, D2, …, D14 9+ 5- D1, D2, D8, D9, D11 [3+, 2-] E=.970 D3, D7, D12, D13 [4+, 0-] D4, D5, D6, D10, D14 [3+, 2-] Gain(S sunny, Humidity) =.970 Gain(S sunny, Temp) =.570 Gain(S sunny, Wind) =.019

17. CSE573 Autumn 1997 17 Final Tree Outlook SunnyRain Humidity Yes Wind Overcast NoYesNoYes HighLowStrongWeak

18. CSE573 Autumn 1997 18 Remaining questions When a node has attributes remaining but no examples? –Constant default value –Based on majority of all examples consistent with the examples classified so far except for the last attribute When a node has examples remaining but no attributes? –Based on majority of all examples that remain unclassified.

19. CSE573 Autumn 1997 19 Some Additional Technical Problems Noise in the data Overfitting Missing values