CS 4100 Artificial Intelligence Prof. C. Hafner Class Notes March 27, 2012

Term Project Presentations Homework 6 is due Tuesday in class (hard copy) We need 4 teams to volunteer to make presentations on April 12 !! The other 5 teams will make presentations on April 17 (last day) Each presentation will be strictly limited to 15 minutes, with 3 minutes for discussion/questions. Make sure your slides/demos load immediately – we do not have time to wait for Google Docs exploration.

Supervised Learning (cont.) Decision Tree Learning (actually, classification learning) – return for further discussion We also consider techniques for evaluating supervised learning systems Perceptrons/Neural Nets Naïve Bayes Classifiers April 3, 5, 10: finish ML, introduce NLP /id3-c45.html#1. /id3-c45.html#1

ID3 and C4.5 Golfing Example: Attributes Decision: Play or Don’t Play

ID3 and C4.5 Golfing Example: Training Data Decision: Play or Don’t Play

Stock Market Example

Table of Entropy values

Review the Algorithm In the case of our golfing example, for the attribute Outlook we have Info(Outlook,T) = 5/14*I(2/5,3/5) + 4/14*I(4/4,0) + 5/14*I(3/5,2/5) = Consider the quantity Gain(X,T) defined as Gain(X,T) = Info(T) - Info(X,T) This represents the difference between the information needed to identify an element of T and the information needed to identify an element of T after the value of attribute X has been obtained, that is, this is the gain in information due to attribute X. In our golfing example, for the Outlook attribute the gain is: Gain(Outlook,T) = Info(T) - Info(Outlook,T) = = If we instead consider the attribute Windy, we find that Info(Windy,T) is and Gain(Windy,T) is Thus Outlook offers a greater informational gain than Windy.

C4.5 Extension Example 1 Notice that in this example two of the attributes have continuous ranges, Temperature and Humidity. ID3 does not directly deal with such cases. We can deal with the case of attributes with continuous ranges as follows: Say that attribute Ci has a continuous range. We examine the values for this attribute in the training set. Say they are, in increasing order, A1, A2,.., Am. Then for each value Aj, j=1,2,..m, we partition the records into those that have Ci values up to and including Aj, and those that have values greater than Aj. For each of these partitions we compute the gain, or gain ratio, and choose the partition that maximizes the gain. This makes Ci a Boolean (or binary) attribute. In our Golfing example, for humidity, if T is the training set, we determine the information for each partition and find the best partition at 75. Then the range for this attribute becomes { 75}. Notice that this method involves a substantial number of computations.

C4.5 Extension Example 2

ID3 and C4.5 ID3 algorithm (we learned last time) is important not because it summarizes what we know, i.e. the training set, but because we hope it will classify correctly new cases. Thus when building classification models one should have both training data to build the model and test data to verify how well it actually works. C4.5 is an extension of ID3 that accounts for unavailable values, continuous attribute value ranges, pruning of decision trees, rule derivation, and so on. C4.5

Perceptrons and Neural Networks: Another Supervised Learning Approach

Perceptron Learning (Supervised) Assign random weights (or set all to 0) Cycle through input data until change < target Let α be the “learning coefficient” For each input: – If perceptron gives correct answer, do nothing – If perceptron says yes when answer should be no, decrease the weights on all units that “fired” by α – If perceptron says no when answer should be yes, increase the weights on all units that “fired” by α

Naive Bayes Classifiers: Our next example of machine learning A supervised learning method Making independence assumption, we can explore a simple subset of Bayesian nets, such that: It is easy to estimate the CPT’s from sample data Uses a technique called “maximum likelihood estimation” – Given a set of correctly classified representative examples – Q: What estimates of conditional probabilities maximize the likelihood of the data that was observed? – A: The estimates that reflect the sample proportions

# Juniors # Non-Juniors were Juniors and were Non-Juniors

Naive Bayes Classifier with multi-valued variables Major: Science, Arts, Social Science Student characteristics: Gender (M,F), Race/Ethnicity (W, B, H, A) International (T/F) What do the conditional probability tables look like??

Theoretical Foundation and Application to Text Classification - thanks Prof. Daphne Koller at Stanford