ID3 and Decision tree by Tuan Nguyen May 2008

ID3 algorithm Is the algorithm to construct a decision tree ID3 and Decision tree Is the algorithm to construct a decision tree Using Entropy to generate the information gain The best value then be selected May 2008

E(S) = -(p+)*log2(p+ ) - (p_ )*log2(p_ ) Entropy ID3 and Decision tree The complete formula for entropy is: E(S) = -(p+)*log2(p+ ) - (p_ )*log2(p_ ) Where p+ is the positive samples Where p_ is the negative samples Where S is the sample of attributions May 2008

Example E(A) = -29/(29+35)*log2(29/(29+35)) – ID3 and Decision tree The Entropy of A1 is computed as the following: A1=? True False [21+, 5-] [8+, 30-] [29+,35-] E(A) = -29/(29+35)*log2(29/(29+35)) – 35/(35+29)log2(35/(35+29)) = 0.9937 E(TRUE) = - 21/(21+5)*log2(21/(21+5)) – 5/(5+21)*log2(5/(5+21)) = 0.7960 E(FALSE) = -8/(8+30)*log2(8/(8+30)) – 30/(30+8)*log2(30/(30+8)) = 0.7426 The Entropy of True: The Entropy of False: May 2008

Information Gain ID3 and Decision tree Gain (Sample, Attributes) or Gain (S,A) is expected reduction in entropy due to sorting S on attribute A So, for the previous example, the Information gain is calculated: G(A1) = E(A1) - (21+5)/(29+35) * E(TRUE) - (8+30)/(29+35) * E(FALSE) = E(A1) - 26/64 * E(TRUE) - 38/64* E(FALSE) = 0.9937– 26/64 * 0.796 – 38/64* 0.7426 = 0.5465 Gain(S,A) = Entropy(S) - vvalues(A) |Sv|/|S| Entropy(Sv) May 2008

The complete example Consider the following table ID3 and Decision tree Consider the following table Day Outlook Temp. Humidity Wind Play Tennis D1 Sunny Hot High Weak No D2 Strong D3 Overcast Yes D4 Rain Mild D5 Cool Normal D6 D7 D8 D9 Cold D10 D11 D12 D13 D14 May 2008

Decision tree We want to build a decision tree for the tennis matches ID3 and Decision tree We want to build a decision tree for the tennis matches The schedule of matches depend on the weather (Outlook, Temperature, Humidity, and Wind) So to apply what we know to build a decision tree based on this table May 2008

Example ID3 and Decision tree Calculating the information gains for each of the weather attributes: For the Wind For the Humidity For the Outlook May 2008

For the Wind Wind Weak Strong [6+, 2-] [3+, 3-] S=[9+,5-] E=0.940 ID3 and Decision tree Wind Weak Strong [6+, 2-] [3+, 3-] S=[9+,5-] E=0.940 Gain(S,Wind): =0.940 - (8/14)*0.811 - (6/14)*1.0 =0.048 May 2008

For the Humidity Humidity High Normal [6+, 1-] S=[9+,5-] E=0.940 ID3 and Decision tree For the Humidity Humidity High Normal [6+, 1-] S=[9+,5-] E=0.940 Gain(S,Humidity) =0.940-(7/14)*0.985 – (7/14)*0.592 =0.151 [3+, 4-] May 2008

For the Outlook Outlook Sunny Rain [2+, 3-] [3+, 2-] S=[9+,5-] E=0.940 ID3 and Decision tree Outlook Sunny Rain [2+, 3-] [3+, 2-] S=[9+,5-] E=0.940 E=0.971 Over cast [4+, 0] E=0.0 Gain(S,Outlook) =0.940-(5/14)*0.971 -(4/14)*0.0 – (5/14)*0.0971 =0.247 May 2008

Complete tree Then here is the complete tree: ID3 and Decision tree Outlook Sunny Overcast Rain Humidity High Normal Wind Strong Weak No Yes [D3,D7,D12,D13] [D8,D9,D11] [D6,D14] [D1,D2] May 2008

Reference: Dr. Lee’s Slides, San Jose State University, Spring 2007 "Building Decision Trees with the ID3 Algorithm", by: Andrew Colin, Dr. Dobbs Journal, June 1996 "Incremental Induction of Decision Trees", by Paul E. Utgoff, Kluwer Academic Publishers, 1989 http://www.cise.ufl.edu/~ddd/cap6635/Fall-97/Short-papers/2.htm http://decisiontrees.net/node/27 May 2008