1. Chapter 12 Machine Learning ID: 116 117 Name: Qun Yu (page 1-33) Kai Zhu (page 34-59) Class: CS267 Fall 2008 Instructor: Dr. T.Y.Lin

2. Introduction Machine Learning is a key part of A.I. research. Rough Set Theory can be used for some problems in Machine Learning. Will discuss 2 cases: 1) Learning from Examples 2) An Imperfect Teacher

3. Case1: Learning from Examples We assume: 2 agents: a knower, a learner Knower ’ s Knowledge set U U is unchanged, will NOT increase during the learning process. It is called CWA(Closed World Assumption) Knower knows everything about U Learner has ability to learn U, in other words, learner knows some attributes of objects in U

4. Example For Case 1 KR-System Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 Attributes of learner’s knowledge : B={a,b,c,d} Attributes of knower’s knowledge : e

5. Example For Case 1 KR-System Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 3 concepts of knower’s knowledge: X 0 ={3,7,10} X 1 ={1,2,4,5,8} X 2 ={6,9} 5 concepts of learner’s knowledge: Y 0 ={1,2} Y 1 ={3,7,10} Y 2 ={4,6} Y={5,9} Y3={5,9} Y 4 ={8}

6. Which Objects are learnable? X 0 ={3,7,10} X 1 ={1,2,4,5,8} X 2 ={6,9} Y 0 ={1,2} Y 1 ={3,7,10} Y 2 ={4,6} Y 3 ={5,9} Y 4 ={8} BX 0 = Y 1 = {3,7,10} = X 0 = BX 0 X 0 is exactly Y-definable and can be learned fully. BX 1 = Y 0  Y 4 = {1,2,8} BX 1 = Y 0  Y 2  Y 3  Y 4 = {1,2,4,5,6,8,9} X 1 is roughly Y-definable, so learner can learn object {1,2,8}, not sure about {4,5,6,9}. BX 2 = Ø BX 1 = Y 2  Y 3 = {4,5,6,9} X 2 is internally Y-indefinable, so it is NOT learnable. Learnable: Learnable: The knower’s knowledge can be expressed in terms of learner’s knowledge.

7. Quality of Learning? BX 0 = Y 1 = {3,7,10} = X 0 = BX 0 BX 1 = Y 0  Y 4 = {1,2,8} BX 1 = Y 0  Y 2  Y 3  Y 4 = {1,2,4,5,6,8,9} BX 2 = Ø BX 1 = Y 2  Y 3 = {4,5,6,9} X0X0 X1X1 X2X2 Positive objects3,7,101,2,8 Ø Border-line objects Ø 3,7,104,5,6,9 Negative objects1,2,4,5,6,8,94,5,6,91,2,3,7,8,10 POS B {e}= POS B (X 0 )  POS B (X 1 )  POS B (X 2 ) = {1,2,3,7,8,10} (6 objects) U={1,2,3,4,5,6,7,8,9,10} (10 objects)  {e}= 6/10 = 0.6

8. Simplifier table … step 1 Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 Group the duplicate rows : Uabcde 3,7,1020010 1,212011 400121 521021 801221 600122 921022

9. Simplifier table … step 2 Remove inconsistent rows : Uabcde 3,7,1020010 1,212011 801221 Uabcde 3,7,1020010 1,212011 400121 521021 801221 600122 921022

10. Simplifier table … step 3 {a}, {b} Find attributes reduct : Uabcde 3,7,1020010 1,212011 801221 Ubcde 3,7,100010 1,22011 81221 Uacde 3,7,102010 1,21011 80221 Uabde 3,7,102010 1,21211 80121 Uabce 3,7,102000 1,21201 80121 Delete a: Delete b: Delete c: Delete d:

11. Simplifier table … step 4 Find value reduct & decision rules: Uae 3,7,1020 1,211 801 Ube 3,7,1000 1,221 811 a2  e0 a1  e1 a0  e1 a2  e0 a1Va0  e1 b0  e0 b2  e1 b1  e1 b0  e0 b2Vb1  e1

12. All objects are necessary in learning process? From the KR-System table, we can see some objects are same by attributes {a,b,c,d,e}: 1 = 2 3 = 7 =10 D 1 ={1,2} D 2 ={3,7,10} Therefore, only one object, either 1 or 2 in D 1 is necessary in learning process. So is D2. 4,5,6,8,9 are necessary. Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010

13. Prove from Decision Rule … (1) Uabcde 112011 320010 400121 521021 600122 801221 921022 Remove objects 2,7 and 10 (keep 1 and 3) as Table1,Table1 will provide the same decision rules. Table1 a 2  e 0 a 1  e 1 a 0  e 1 b 0  e 0 b 2  e 1 b 1  e 1 OR

14. Prove from Decision Rule … (2) Uabcde 212011 400121 521021 600122 801221 921022 1020010 Remove objects 1,3 and 7 (keep 2 and 10)as Table 2,Table2 will provide the same decision rules. Table 2 a 2  e 0 a 1  e 1 a 0  e 1 b 0  e 0 b 2  e 1 b 1  e 1 OR

15. Prove from Decision Rule … (3) Uabcde 112011 212011 320010 521021 600122 720010 921022 1020010 Remove objects 4 and 8 as Table3, The whole new decision algorithm will be: Table 3 a 2  e 0 a 1  e 1 a 0  e 2 Now concept X 2 Now is internally definable. Object 6 is learnable now. So decision algorithm changed. b 2 c 0  e 1 b 0 c 0  e 0 b 0 c 1  e 2 b 2 d 1  e 1 b 0 d 1  e 0 b 0 d 2  e 2 OR

16. Prove from Decision Rule … (4) Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 1020010 Table 4 Remove objects 9 as Table 4, The whole new decision algorithm will be: b 0  e 0 b 2  e 1 a 0 b 1  e 1 a 2 b 1  e 1 Now object 5 is positive object of X 1. So decision algorithm changed. a 2 d 1  e 0 a 1 d 1  e 1 a 2 d 2  e 1 a 0 d 2  e 1 OR

17. Case2: An Imperfect Teacher We assume: 2 agents: a knower, a learner Knower ’ s Knowledge set U U is unchanged, will NOT increase during the learning process. It is called CWA(Closed World Assumption) Knower knows everything about U Learner has ability to learn U, in other words, learner knows some attributes of objects in U

18. Example 1 For Case 2 KR-System Uabc 102+ 201+ 310+ 4010 5100 611- 721- 801- 910- Attributes of learner’s knowledge : B={a,b} Attributes of knower’s knowledge : c Attribute value 0: knower can’t classify this object.

19. Example 1 For Case 2 KR-System 3 concepts of knower’s knowledge: X 0 ={4,5}  ignorance region X + ={1,2,3} X - ={6,7,8,9} X*=X +  X -  competence region 5 concepts of learner’s knowledge: Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y={6} Y3={6} Y 4 ={7} Uabc 102+ 201+ 310+ 4010 5100 611- 721- 801- 910-

20. Can the learner be able to discover the ignorance region? BX + = Y 0 = {1} BX + = Y 0  Y 1  Y 2 = {1,2,3,4,5,8,9} X + is roughly Y-definable, so learner can learn object {1}, not sure about {2,3,4,5,8,9}. BX - = Y 3  Y 4 = {6,7} BX - = Y 1  Y 2  Y 3  Y 4 = {2,3,4,5,6,7,8,9} X - is roughly Y-definable, so learner can learn object {6,7}, not sure about {2,3,4,5,8,9}. BX 0 = Ø BX 0 = Y 1  Y 2 = {2,3,4,5,8,9} X 0 is internally Y-indefinable, so it is NOT learnable. X 0 ={4,5}  ignorance region X + ={1,2,3} X - ={6,7,8,9} X*=X +  X -  competence region Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y 3 ={6} Y 4 ={7}  Learner can’t discover the ignorance region X 0

21. The ignorance region influences the learner ’ s ability to learn? In this example, the answer is NO, because the ignorance region(X0) is not learnable. Hence, we can prove it: BN B (X*)= BN B (X + )  BN B (X - ) BX + = Y 0 = {1} BX + = Y 0  Y 1  Y 2 = {1,2,3,4,5,8,9} BX - = Y 3  Y 4 = {6,7} BX - = Y 1  Y 2  Y 3  Y 4 = {2,3,4,5,6,7,8,9} X*=X +  X -  competence region X 0 ={4,5}  ignorance region X* put 4 into X + X* put 4 into X - Positive objects 1,6,7 Border-line objects 2,3,4,5,8,9 Negative objects ØØØ Border-line region of the competence region X* remain unchanged, therefore, it doesn't matter knower knows it or NOT.

22. Put 4 into X + X + ={1,2,3,4} X - ={6,7,8,9} X 0 ={5} BX + = Y 0 = {1} 2,3,4,5,8,9 BX + = Y 0  Y 1  Y 2 = {1,2,3,4,5,8,9} BX - = Y 3  Y 4 = {6,7} 2,3,4,58,9 BX - = Y 1  Y 2  Y 3  Y 4 = {2,3,4,5,6,7,8,9} BN B (X*)= BN B (X + )  BN B (X - ) ={2,3,4,5,8,9}  {2,3,4,5,8,9} ={2,3,4,5,8,9} X 0 ={4,5}  ignorance region X + ={1,2,3} X - ={6,7,8,9} X*=X +  X -  competence region Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y 3 ={6} Y 4 ={7} Calculation Sheet: Border-line objects

23. Simplifier table … step 1 Remove inconsistent rows : Uabc 102+ 611- 721- Uabc 102+ 201+ 310+ 4010 5100 611- 721- 801- 910-  Learner can’t discover the ignorance region {4,5} and the ignorance region won’t affect the learning process.

24. Simplifier table … step 2 {a}, {b} Find attributes reduct : Delete a: Delete b: Uabc 102+ 611- 721- Uac 10+ 61- 72- Ubc 12+ 61- 71-

25. Simplifier table … step 3 Find value reduct & decision rules: a0  c+ a1  c- a2  c- b2  c+ b1  c- Uac 10+ 61- 72- Ubc 12+ 61- 71-

26. Example 2 For Case 2 KR-System Uabc 102+ 201+ 310+ 401+ 5100 6110 721- 801- 910- Attributes of learner’s knowledge : B={a,b} Attributes of knower’s knowledge : c Attribute value 0: knower can’t classify this object.

27. Example 2 For Case 2 KR-System 3 concepts of knower’s knowledge: X 0 ={5,6}  ignorance region X + ={1,2,3,4} X - ={7,8,9} X*=X +  X -  competence region 5 concepts of learner’s knowledge: Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y={6} Y3={6} Y 4 ={7} Uabc 102+ 201+ 310+ 401+ 5100 6110 721- 801- 910-

28. Can the learner be able to discover the ignorance region? BX + = Y 0 = {1} BX + = Y 0  Y 1  Y 2 = {1,2,3,4,5,8,9} X + is roughly Y-definable, so learner can learn object {1}, not sure about {2,3,4,5,8,9}. BX - = Y 4 = {7} BX - = Y 1  Y 2  Y 4 = {2,3,4,5,7,8,9} X - is roughly Y-definable, so learner can learn object {7}, not sure about {2,3,4,5,8,9}. BX 0 = Y 3 = {6} BX 0 = Y 2  Y 3 = {3,5,6,9} X 0 is roughly Y-definable, so learner can learn object {6}, not sure about {3,5,9}. X + ={1,2,3,4} X - ={7,8,9} X*=X +  X -  competence region X 0 ={5,6}  ignorance region Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y 3 ={6} Y 4 ={7}  Learner can discover the ignorance region X 0

29. The ignorance region influences the learner ’ s ability to learn? In this example, the answer is YES, because the ignorance region(X0) is roughly learnable, object 6 is important. Hence, we can prove it: BX + = Y 0 = {1} BX + = Y 0  Y 1  Y 2 = {1,2,3,4,5,8,9} BX - = Y 4 = {7} BX - = Y 1  Y 2  Y 4 = {2,3,4,5,7,8,9} X*=X +  X -  competence region X 0 ={5,6}  ignorance region X* put 6 into X + X* put 5 into X - Positive objects 1, 71,6,71, 7 Border-line objects 2,3,4,5,8,9 Negative objects 6Ø6 Object 6 will fall in positive region if knower knows it (move from X 0 to X + ). Therefore, X 0 affects the learning process. BN B (X*)= BN B (X + )  BN B (X - )

30. Put 6 into X + X + ={1,2,3,4,6} X - ={7,8,9} X 0 ={5} BX + = Y 0 = {1,6} 2,3,4,5,8,9 BX + = Y 0  Y 1  Y 2  Y 3 = {1,2,3,4,5,6,8,9} BX - = Y 4 = {7} 2,3,4,58,9 BX - = Y 1  Y 2  Y 4 = {2,3,4,5,7,8,9} POS B (X*)= POS B (X + )  POS B (X - ) ={1,6}  {7} ={1,6,7} X 0 ={5,6}  ignorance region X + ={1,2,3,4} X - ={7,8,9} X*=X +  X -  competence region Y 0 ={1} Y 1 ={2,4,8} Y 2 ={3,5,9} Y 3 ={6} Y 4 ={7} Calculation Sheet: Positive objects

31. Simplifier table … step 1 Remove inconsistent rows : Uabc 102+ 6110 721-  Object 6 is learnable, therefore learner can discover the ignorance region {4,5} and the ignorance region will affect the learning process. Uabc 102+ 201+ 310+ 401+ 5100 6110 721- 801- 910-

32. Simplifier table … step 2 {a} Find attributes reduct : Delete a: Delete b: Uabc 102+ 6110 721- Uac 10+ 610 72- Ubc 12+ 610 71- A is indispensable

33. Simplifier table … step 3 Find value reduct & decision rules: a0  c+ a1  c0 a2  c- Uac 10+ 610 72- Object 6 is learnable, so learner can discover the knower’s ignorance.

34. Chapter 12 Machine Learning Inductive Learning Presented by: Kai Zhu Professor: Dr. T.Y. Lin Class ID: 117

35. In the pervious chapters, we assumed that the set of instances U is constant and unchanged during the learning process. In any real life situations however this is not the case and new instances can be added to the set U.

36. Example Lets consider the K-R system given in Example 1 in the pervious section in chapter 12

37. Table 9 Table 9 Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010

38. Consistent table: Uabcde 112011 212011 320010 720010 801221 1020010 Inconsistent table: Uabcde 400121 521021 600122 921022 (e)=card(1,2,3,7,8,10)/card(1,2,3,4,5,6,7,8,9,10)=6/10=0.6

39. After remove duplicate and inconsistent Uabcde 112011 220010 301221

40. After computing, get core and reduct values table Uabcde 1----1 2----0 3----1 Uabcde 1(1)1xxx1 1(2)x2xx1 2(1)2xxx0 2(2)x0xx0 3(1)0xxx1 3(2)x1xx1 3(3)xx2x1 3(4)xxx21 Core table Reduct table

41. From reduct table, there are 16 Simplified Tables. One of them : corresponding decision algorithms: Uabcde 1(1)1xxx1 2(1)2xxx0 3(1)0xxx1 a1  e1 a2  e0 a0  e1

42. Table 10 Table 10 Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 1101221

43. Consistent table: Inconsistent table: (e)=card(1,2,3,7,8,10,11)/card(1,2,3,4,5,6,7,8,9,10,11)=7/11=0.636 Uabcde 112011 212011 320010 720010 801221 1020010 1101221 Uabcde 400121 521021 600122 921022

44. After remove duplicate and inconsistent Uabcde 112011 220010 301221

45. After computing, get core and reduct values table Uabcde 1----1 2----0 3----1 Uabcde 1(1)1xxx1 1(2)x2xx1 2(1)2xxx0 2(2)x0xx0 3(1)0xxx1 3(2)x1xx1 3(3)xx2x1 3(4)xxx21 Core table Reduct table

46. From reduct table, there are 16 Simplified Tables. One of them : corresponding decision algorithms: Uabcde 1(1)1xxx1 2(1)2xxx0 3(1)0xxx1 a1  e1 a2  e0 a0  e1

47. It is obvious that in table 10 the new instance does not change the decision algorithm, that means that the learned concepts will remain the same. But the quality of learning  (e) changed. (from 0.6 to 0.636)

48. Table 11 Table 11 Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 1112010

49. Consistent table: Inconsistent table: (e)=card(3,7,8,10)/card(1,2,3,4,5,6,7,8,9,10,11)=4/11=0.363 Uabcde 320010 720010 801221 1020010 Uabcde 112011 212011 400121 521021 600122 921022 1112010

50. After remove duplicate and inconsistent Uabcde 120010 201221

51. After computing, get core and reduct values table Core table Reduct table abcde ----0 ----1 Uabcde 1(1)2xxx0 1(2)x0xx0 1(3)xx0x0 1(4)xxx10 2(1)0xxx1 2(2)x1xx1 2(3)xx2x1 2(4)xxx21

52. From reduct table, there are 16 Simplified Tables. One of them : corresponding decision algorithms: Uabcde 1(1)2xxx0 2(1)0xxx1 a2  e0 a0  e1 (Compare to a1  e1 a2  e0 a0  e1)

53. Table 12 Table 12 Uabcde 112011 212011 320010 400121 521021 600122 720010 801221 921022 1020010 1110013

54. Consistent table: Inconsistent table: (e)=card(1,2,3,7,8,10,11)/card(1,2,3,4,5,6,7,8,9,10)=6/10=0.636 Uabcde 112011 212011 320010 720010 801221 1020010 1110013 Uabcde 400121 521021 600122 921022

55. After remove duplicate and inconsistent Uabcde 112011 220010 301221 410013

56. After computing, get core and reduct values table Core table Reduct table Uabcde 1-2--1 22---0 3----1 410--3 Uabcde 1(1)x2xx1 2(1)2xxx0 3(1)0xxx1 3(2)x1xx1 3(3)xx2x1 3(4)xxx21 4(1)10xx3

57. From reduct table, there are 4 Simplified Tables. One of them : corresponding decision algorithms: Uabcde 1(1)x2xx1 2(1)2xxx0 3(1)0xxx1 4(1)10xx3 b2  e1 a2  e0 a0  e1 a1b0  e3 (Compare to a1  e1 a2  e0 a0  e1)

58. To sum up, as we have seen from the above examples, adding a new instance to the universe we could face three possibilities: 1. The new instance confirms actual knowledge. (table10) 2. The new instance contradicts the actual knowledge. (table11) 3. The new instance is a completely new case.(table12)

59. Thank you