1. COMP53311 Classification Prepared by Raymond Wong The examples used in Decision Tree are borrowed from LW Chan ’ s notes Presented by Raymond Wong raywong@cse

2. COMP53312 Classification root child=yeschild=no Income=high Income=low 100% Yes 0% No 100% Yes 0% No 0% Yes 100% No Decision tree RaceIncomeChildInsurance whitehighno? Suppose there is a person.

3. COMP53313 Classification root child=yeschild=no Income=high Income=low 100% Yes 0% No 100% Yes 0% No 0% Yes 100% No Decision tree RaceIncomeChildInsurance whitehighno? Suppose there is a person. RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Training set Test set

4. COMP53314 Applications Insurance According to the attributes of customers, Determine which customers will buy an insurance policy Marketing According to the attributes of customers, Determine which customers will buy a product such as computers Bank Loan According to the attributes of customers, Determine which customers are “ risky ” customers or “ safe ” customers

5. COMP53315 Applications Network According to the traffic patterns, Determine whether the patterns are related to some “ security attacks ” Software According to the experience of programmers, Determine which programmers can fix some certain bugs

6. COMP53316 Same/Difference Classification Clustering

7. COMP53317 Classification Methods Decision Tree Bayesian Classifier Nearest Neighbor Classifier

8. COMP53318 Decision Trees ID3 C4.5 CART Iterative Dichotomiser Classification And Regression Trees Classification

9. COMP53319 Entropy Example 1 Consider a random variable which has a uniform distribution over 32 outcomes To identify an outcome, we need a label that takes 32 different values. Thus, 5 bit strings suffice as labels

10. COMP533110 Entropy Entropy is used to measure how informative is a node. If we are given a probability distribution P = (p 1, p 2, …, p n ) then the Information conveyed by this distribution, also called the Entropy of P, is: I(P) = - (p 1 x log p 1 + p 2 x log p 2 + … + p n x log p n ) All logarithms here are in base 2.

11. COMP533111 Entropy For example, If P is (0.5, 0.5), then I(P) is 1. If P is (0.67, 0.33), then I(P) is 0.92, If P is (1, 0), then I(P) is 0. The entropy is a way to measure the amount of information. The smaller the entropy, the more informative we have.

12. COMP533112 Entropy RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - ½ log ½ - ½ log ½ = 1 Info(T black ) = - ¾ log ¾ - ¼ log ¼ For attribute Race, Info(T white ) = - ¾ log ¾ - ¼ log ¼ Info(Race, T) = ½ x Info(T black ) + ½ x Info(T white ) Gain(Race, T) = Info(T) – Info(Race, T)= 1 – 0.8113 = 0.1887 For attribute Race, Gain(Race, T) = 0.1887 = 0.8113

13. COMP533113 Entropy RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - ½ log ½ - ½ log ½ = 1 Info(T low ) = - 1/3 log 1/3 – 2/3 log 2/3 For attribute Income, Info(T high ) = - 1 log 1 – 0 log 0 Info(Income, T) = ¼ x Info(T high ) + ¾ x Info(T low ) Gain(Income, T) = Info(T) – Info(Income, T)= 1 – 0.6887 = 0.3113 For attribute Race, Gain(Race, T) = 0.1887 For attribute Income, Gain(Income, T) = 0.3113 = 0.9183 = 0 = 0.6887

14. COMP533114 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - ½ log ½ - ½ log ½ = 1 Info(T yes ) = - 1 log 1 – 0 log 0 For attribute Child, Info(T no ) = - 1/5 log 1/5 – 4/5 log 4/5 Info(Child, T) = 3/8 x Info(T yes ) + 5/8 x Info(T no ) Gain(Child, T) = Info(T) – Info(Child, T)= 1 – 0.4512 = 0.5488 For attribute Race, Gain(Race, T) = 0.1887 For attribute Income, Gain(Income, T) = 0.3113 For attribute Child, Gain(Child, T) = 0.5488 = 0.7219 = 0 = 0.4512 1 2 3 4 5 6 7 8 root child=yeschild=no {2, 3, 4} {1, 5, 6, 7, 8} Insurance: 3 Yes; 0 No Insurance: 1 Yes; 4 No 100% Yes 0% No 20% Yes 80% No

15. COMP533115 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - 1/5 log 1/5 – 4/5 log 4/5 = 0.7219 Info(T black ) = - ¼ log ¼ – ¾ log ¾ For attribute Race, Info(T white ) = - 0 log 0 – 1 log 1 Info(Race, T) = 4/5 x Info(T black ) + 1/5 x Info(T white ) Gain(Race, T) = Info(T) – Info(Race, T)= 0.7219 – 0.6490 = 0.0729 For attribute Race, Gain(Race, T) = 0.0729 = 0 = 0.8113 = 0.6490 1 2 3 4 5 6 7 8 root child=yeschild=no {2, 3, 4} {1, 5, 6, 7, 8} Insurance: 3 Yes; 0 No Insurance: 1 Yes; 4 No 100% Yes 0% No 20% Yes 80% No

16. COMP533116 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - 1/5 log 1/5 – 4/5 log 4/5 = 0.7219 Info(T high ) = - 1 log 1 – 0 log 0 For attribute Income, Info(T low ) = - 0 log 0 – 1 log 1 Info(Income, T) = 1/5 x Info(T high ) + 4/5 x Info(T low ) Gain(Income, T) = Info(T) – Info(Income, T)= 0.7219 – 0 = 0.7219 For attribute Race, Gain(Race, T) = 0.0729 = 0 1 2 3 4 5 6 7 8 root child=yeschild=no {2, 3, 4} {1, 5, 6, 7, 8} Insurance: 3 Yes; 0 No Insurance: 1 Yes; 4 No For attribute Income, Gain(Income, T) = 0.7219 100% Yes 0% No 20% Yes 80% No

17. COMP533117 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - 1/5 log 1/5 – 4/5 log 4/5 = 0.7219 Info(T high ) = - 1 log 1 – 0 log 0 For attribute Income, Info(T low ) = - 0 log 0 – 1 log 1 Info(Income, T) = 1/5 x Info(T high ) + 4/5 x Info(T low ) Gain(Income, T) = Info(T) – Info(Income, T)= 0.7219 – 0 = 0.7219 For attribute Race, Gain(Race, T) = 0.0729 = 0 1 2 3 4 5 6 7 8 root child=yeschild=no For attribute Income, Gain(Income, T) = 0.7219 Income=high Income=low 100% Yes 0% No 20% Yes 80% No {1} {5, 6, 7, 8} Insurance: 1 Yes; 0 No Insurance: 0 Yes; 4 No 100% Yes 0% No 0% Yes 100% No

18. COMP533118 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno 1 2 3 4 5 6 7 8 root child=yeschild=no Income=high Income=low 100% Yes 0% No 100% Yes 0% No 0% Yes 100% No Decision tree RaceIncomeChildInsurance whitehighno? Suppose there is a new person.

19. COMP533119 RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno 1 2 3 4 5 6 7 8 root child=yeschild=no Income=high Income=low 100% Yes 0% No 100% Yes 0% No 0% Yes 100% No Decision tree Termination Criteria? e.g., height of the tree e.g., accuracy of each node

20. COMP533120 Decision Trees ID3 C4.5 CART

21. COMP533121 C4.5 ID3 Impurity Measurement Gain(A, T) = Info(T) – Info(A, T) C4.5 Impurity Measurement Gain(A, T) = (Info(T) – Info(A, T))/SplitInfo(A) where SplitInfo(A) = -  v  A p(v) log p(v)

22. COMP533122 Entropy RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - ½ log ½ - ½ log ½ = 1 For attribute Race, Gain(Race, T) = (Info(T) – Info(Race, T))/SplitInfo(Race)= (1 – 0.8113)/1 = 0.1887 For attribute Race, Gain(Race, T) = 0.1887 Info(T black ) = - ¾ log ¾ - ¼ log ¼ Info(T white ) = - ¾ log ¾ - ¼ log ¼ Info(Race, T) = ½ x Info(T black ) + ½ x Info(T white ) = 0.8113 SplitInfo(Race) = - ½ log ½ - ½ log ½ = 1

23. COMP533123 Entropy RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = - ½ log ½ - ½ log ½ = 1 For attribute Income, Gain(Income, T)= (Info(T) – Info(Income, T))/SplitInfo(Income)= (1 – 0.6887)/0.8113 = 0.3837 For attribute Race, Gain(Race, T) = 0.1887 For attribute Income, Gain(Income, T) = 0.3837 Info(T low ) = - 1/3 log 1/3 – 2/3 log 2/3 Info(T high ) = - 1 log 1 – 0 log 0 Info(Income, T) = ¼ x Info(T high ) + ¾ x Info(T low ) = 0.9183 = 0 = 0.6887 SplitInfo(Income) = - 2/8 log 2/8 – 6/8 log 6/8= 0.8113 For attribute Child, Gain(Child, T) = ?

24. COMP533124 Decision Trees ID3 C4.5 CART

25. COMP533125 CART Impurity Measurement Gini I(P) = 1 –  j p j 2

26. COMP533126 Gini RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = 1 – ( ½ ) 2 – ( ½ ) 2 = ½ Info(T black ) = 1 – ( ¾ ) 2 – ( ¼ ) 2 For attribute Race, Info(T white ) = 1 – ( ¾ ) 2 – ( ¼ ) 2 Info(Race, T) = ½ x Info(T black ) + ½ x Info(T white ) Gain(Race, T) = Info(T) – Info(Race, T)= ½ – 0.375 = 0.125 For attribute Race, Gain(Race, T) = 0.125 = 0.375

27. COMP533127 Gini RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno Info(T) = 1 – ( ½ ) 2 – ( ½ ) 2 = ½ Info(T low ) = 1 – (1/3) 2 – (2/3) 2 For attribute Income, Info(T high ) = 1 – 1 2 – 0 2 Info(Income, T) = 1/4 x Info(T high ) + 3/4 x Info(T low ) Gain(Income, T) = Info(T) – Info(Race, T)= ½ – 0.333 = 0.167 For attribute Race, Gain(Race, T) = 0.125 = 0.444 = 0.333 = 0 For attribute Income, Gain(Race, T) = 0.167 For attribute Child, Gain(Child, T) = ?

28. COMP533128 Classification Methods Decision Tree Bayesian Classifier Nearest Neighbor Classifier

29. COMP533129 Bayesian Classifier Na ï ve Bayes Classifier Bayesian Belief Networks

30. COMP533130 Na ï ve Bayes Classifier Statistical Classifiers Probabilities Conditional probabilities

31. COMP533131 Na ï ve Bayes Classifier Conditional Probability A: a random variable B: a random variable P(A | B) = P(AB) P(B)

32. COMP533132 Na ï ve Bayes Classifier Bayes Rule A : a random variable B: a random variable P(A | B) = P(B|A) P(A) P(B)

33. COMP533133 Na ï ve Bayes Classifier Independent Assumption Each attribute are independent e.g., P(X, Y, Z | A) = P(X | A) x P(Y | A) x P(Z | A) RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno

34. COMP533134 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = P(Race = white | Yes) x P(Income = high | Yes) x P(Child = no | Yes) = ¾ x ½ x ¼ = 0.09375 P(Race = white, Income = high, Child = no| No) = P(Race = white | No) x P(Income = high | No) x P(Child = no | No) = ¼ x 0 x 1 = 0

35. COMP533135 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = P(Race = white | No) x P(Income = high | No) x P(Child = no | No) = ¼ x 0 x 1 = 0

36. COMP533136 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = P(Race = white | No) x P(Income = high | No) x P(Child = no | No) = ¼ x 0 x 1 = 0

37. COMP533137 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = 0

38. COMP533138 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = 0

39. COMP533139 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = 0 P(Yes | Race = white, Income = high, Child = no) = P(Race = white, Income = high, Child = no| Yes) P(Yes) P(Race = white, Income = high, Child = no) = 0.09375 x 0.5 P(Race = white, Income = high, Child = no) = 0.046875 P(Race = white, Income = high, Child = no) P(Yes | Race = white, Income = high, Child = no) = 0.046875 P(Race = white, Income = high, Child = no)

40. COMP533140 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. P(Race = white, Income = high, Child = no| Yes) Na ï ve Bayes Classifier = 0.09375 P(Race = white, Income = high, Child = no| No) = 0 P(No | Race = white, Income = high, Child = no) = P(Race = white, Income = high, Child = no| No) P(No) P(Race = white, Income = high, Child = no) = 0 x 0.5 P(Race = white, Income = high, Child = no) = 0 P(Yes | Race = white, Income = high, Child = no) = 0.046875 P(Race = white, Income = high, Child = no) P(No | Race = white, Income = high, Child = no) = 0 P(Race = white, Income = high, Child = no)

41. COMP533141 Na ï ve Bayes Classifier RaceIncomeChildInsurance blackhighnoyes whitehighyes whitelowyes whitelowyes blacklowno blacklowno blacklowno whitelowno P(Race = black | Yes) = ¼ For attribute Race, P(Race = white | Yes) = ¾ P(Race = black | No) = ¾ P(Race = white | No) = ¼ P(Income = high | Yes) = ½ For attribute Income, P(Income = low | Yes) = ½ P(Income = high | No) = 0 P(Income = low | No) = 1 P(Child = yes | Yes) = ¾ For attribute Child, P(Child = no | Yes) = ¼ P(Child = yes | No) = 0 P(Child = no | No) = 1 Insurance = Yes P(Yes) = ½ P(No) = ½ RaceIncomeChildInsurance whitehighno? Suppose there is a new person. Na ï ve Bayes Classifier P(Yes | Race = white, Income = high, Child = no) = 0.046875 P(Race = white, Income = high, Child = no) P(No | Race = white, Income = high, Child = no) = 0 P(Race = white, Income = high, Child = no) Since P(Yes | Race = white, Income = high, Child = no) > P(No | Race = white, Income = high, Child = no). we predict the following new person will buy an insurance. RaceIncomeChildInsurance whitehighno?

42. COMP533142 Bayesian Classifier Na ï ve Bayes Classifier Bayesian Belief Networks

43. COMP533143 Bayesian Belief Network Na ï ve Bayes Classifier Independent Assumption Bayesian Belief Network Do not have independent assumption

44. COMP533144 Bayesian Belief Network ExerciseDietHeartburnBlood PressureChest PainHeart Disease YesHealthyNoHighYesNo UnhealthyYesLowYesNo HealthyYesHighNoYes ……………… Yes/No Healthy/ Unhealthy Yes/No High/Low Some attributes are dependent on other attributes. e.g., doing exercises may reduce the probability of suffering from Heart Disease Exercise (E) Heart Disease

45. COMP533145 Bayesian Belief Network Exercise (E)Diet (D) Heart Disease (HD)Heartburn (Hb) Blood Pressure (BP)Chest Pain (CP) E = Yes 0.7 D = Healthy 0.25 HD=Yes E=Yes D=Healthy 0.25 E=Yes D=Unhealthy 0.45 E=No D=Healthy 0.55 E=No D=Unhealthy 0.75 CP=Yes HD=Yes Hb=Yes 0.8 HD=Yes Hb=No 0.6 HD=No Hb=Yes 0.4 HD=No Hb=No 0.1 BP=High HD=Yes0.85 HD=No0.2 Hb=Yes D=Healthy0.85 D=Unhealthy0.2

46. COMP533146 Bayesian Belief Network Let X, Y, Z be three random variables. X is said to be conditionally independent of Y given Z if the following holds. P(X | Y, Z) = P(X | Z) Lemma: If X is conditionally independent of Y given Z, P(X, Y | Z) = P(X | Z) x P(Y | Z) ?

47. COMP533147 Bayesian Belief Network Exercise (E)Diet (D) Heart Disease (HD)Heartburn (Hb) Blood Pressure (BP)Chest Pain (CP) Let X, Y, Z be three random variables. X is said to be conditionally independent of Y given Z if the following holds. P(X | Y, Z) = P(X | Z) e.g., P(BP = High | HD = Yes, D = Healthy) = P(BP = High | HD = Yes) e.g., P(BP = High | HD = Yes, CP=Yes) = P(BP = High | HD = Yes) “ BP = High ” is conditionally independent of “ D = Healthy ” given “ HD = Yes ” “ BP = High ” is conditionally independent of “ CP = Yes ” given “ HD = Yes ” Property: A node is conditionally independent of its non-descendants if its parents are known.

48. COMP533148 Bayesian Belief Network ExerciseDietHeartburnBlood PressureChest PainHeart Disease YesHealthyNoHighYesNo UnhealthyYesLowYesNo HealthyYesHighNoYes ……………… Suppose there is a new person and I want to know whether he is likely to have Heart Disease. ExerciseDietHeartburnBlood PressureChest PainHeart Disease ?????? Yes/No Healthy/ Unhealthy Yes/No High/Low ExerciseDietHeartburnBlood PressureChest PainHeart Disease ???High?? ExerciseDietHeartburnBlood PressureChest PainHeart Disease YesHealthy?High??

49. COMP533149 Bayesian Belief Network Suppose there is a new person and I want to know whether he is likely to have Heart Disease. ExerciseDietHeartburnBlood PressureChest PainHeart Disease ?????? P(HD = Yes) =  x  {Yes, No}  y  {Healthy, Unhealthy} P(HD=Yes|E=x, D=y) x P(E=x, D=y) =  x  {Yes, No}  y  {Healthy, Unhealthy} P(HD=Yes|E=x, D=y) x P(E=x) x P(D=y) = 0.25 x 0.7 x 0.25 + 0.45 x 0.7 x 0.75 + 0.55 x 0.3 x 0.25 + 0.75 x 0.3 x 0.75 = 0.49 P(HD = No) = 1- P(HD = Yes) = 1- 0.49 = 0.51

50. COMP533150 Bayesian Belief Network Suppose there is a new person and I want to know whether he is likely to have Heart Disease. ExerciseDietHeartburnBlood PressureChest PainHeart Disease ???High?? P(BP = High) =  x  {Yes, No} P(BP = High|HD=x) x P(HD = x) = 0.85x0.49 + 0.2x0.51 = 0.5185 P(HD = Yes|BP = High) = P(BP = High|HD=Yes) x P(HD = Yes) P(BP = High) = 0.85 x 0.49 0.5185 = 0.8033 P(HD = No|BP = High) = 1 – P(HD = Yes|BP = High) = 1 – 0.8033 = 0.1967

51. COMP533151 Bayesian Belief Network Suppose there is a new person and I want to know whether he is likely to have Heart Disease. ExerciseDietHeartburnBlood PressureChest PainHeart Disease YesHealthy?High?? P(HD = Yes | BP = High, D = Healthy, E = Yes) = P(BP = High | HD = Yes, D = Healthy, E = Yes) P(BP = High | D = Healthy, E = Yes) x P(HD = Yes|D = Healthy, E = Yes) P(BP = High|HD = Yes) P(HD = Yes|D = Healthy, E = Yes)  x  {Yes, No} P(BP=High|HD=x) P(HD=x|D=Healthy, E=Yes) = 0.85x0.25 0.85x0.25 + 0.2x0.75 = = 0.5862 P(HD = No | BP = High, D = Healthy, E = Yes) = 1- P(HD = Yes | BP = High, D = Healthy, E = Yes) = 1-0.5862 = 0.4138

52. COMP533152 Classification Methods Decision Tree Bayesian Classifier Nearest Neighbor Classifier

53. COMP533153 Nearest Neighbor Classifier ComputerHistory 10040 9045 2095 …… Computer History

54. COMP533154 Nearest Neighbor Classifier ComputerHistoryBuy Book? 10040No (-) 9045Yes (+) 2095Yes (+) ……… Computer History + + + -- - + - - - +

55. COMP533155 Nearest Neighbor Classifier ComputerHistoryBuy Book? 10040No (-) 9045Yes (+) 2095Yes (+) ……… Computer History + + + -- - + - - - + Suppose there is a new person ComputerHistoryBuy Book? 9535? Nearest Neighbor Classifier: Step 1: Find the nearest neighbor Step 2: Use the “ label ” of this neighbor

56. COMP533156 Nearest Neighbor Classifier ComputerHistoryBuy Book? 10040No (-) 9045Yes (+) 2095Yes (+) ……… Computer History + + + -- - + - - - + Suppose there is a new person ComputerHistoryBuy Book? 9535? k-Nearest Neighbor Classifier: Step 1: Find k nearest neighbors Step 2: Use the majority of the labels of the neighbors