1. KBS development life cycle Validation Uncertainty KBS development life cycle Validation Uncertainty

2. Dr. R. Weber INFO612 KBS development life cycle Validation Uncertainty KBS development life cycle Validation Uncertainty

3. Copyright Rosina Weber ES Development Life Cycle ES Development Life Cycle Planning Planning Knowledge Definition Knowledge Definition Knowledge Design Knowledge Design Code and Checkout Code and Checkout Knowledge Verification Knowledge Verification System Evaluation System Evaluation

4. Copyright Rosina Weber Planning feasibility assessment feasibility assessment resource management resource management task phasing task phasing schedules schedules preliminary functional layout (what) preliminary functional layout (what) high-level requirements (how) high-level requirements (how)

5. Copyright Rosina Weber Knowledge Definition knowledge source identification and selection source identification, importance, availability, selection source identification, importance, availability, selection knowledge acquisition, analysis, & extraction acquisition strategy acquisition strategy knowledge identification and classification knowledge identification and classification functional layout functional layout control flow control flow user’s manual user’s manual requirements specification requirements specification knowledge baseline knowledge baseline

6. Copyright Rosina Weber Knowledge Design knowledge representation choose the most appropriate knowledge representation formalism and then choose the tool choose the most appropriate knowledge representation formalism and then choose the tool control structure internal fact structure preliminary user interface initial test plan design structure implementation strategy detailed user interface design specifications and report detailed test plan

7. Copyright Rosina Weber Code and Checkout coding testing source listings user’s manual installation/operations guide system description document

8. Copyright Rosina Weber Knowledge Verification formal tests test analysis incorrect incomplete inconsistent recommendations

9. Copyright Rosina Weber System Evaluation-Validation demonstrate the system serves its purposes efficiently and effectively comparing to other systems comparing to alternate methods comparing to humans maintenance

10. Copyright Rosina Weber Planning Planning Knowledge acquisition Knowledge acquisition Knowledge engineering: design and implementation Knowledge engineering: design and implementation Situation assessment Situation assessment Retrieve Retrieve Revise Revise Review Review Retain Retain System Evaluation System Evaluation Maintain Maintain CBR Development Life Cycle CBR Development Life Cycle

11. Copyright Rosina Weber CBR design and implementation Situation assessment Situation assessment Retrieve Retrieve Revise Revise Review Review Retain Retain (Validation) (Validation) Maintenance design Maintenance design

12. Copyright Rosina Weber case base Design decisions in CBR systems (i) Which are the cases? What is the task? How will the case base be organized? How will the cases be represented? Which will be the indexing vocabulary? What is the task? How will the case base be organized?

13. Copyright Rosina Weber How will new cases be input? How to perform retrieval? Identify features Initially match (similarity assessment) Search Select Retrieval input problem initial solutions Design decisions in CBR systems (ii)

14. Copyright Rosina Weber How to implement reuse? From Select or with a combination? How to display the proposed solution? solution Reuse proposed initial solutions Design decisions in CBR systems (iii)

15. Copyright Rosina Weber Is the proposed solution good? How to determine and find what to adapt? Where is adaptation knowledge? solution Revise proposed confirmed solution case repair case adaptation Design decisions in CBR systems (iv)

16. Copyright Rosina Weber Is it the type of task that it is worth learning? Index new case before retain. Retain. Retain confirmed solution case base Design decisions in CBR systems (ii)

17. Copyright Rosina Weber Validation refers to establishing the effectiveness of a system in light of its intended purposes Validation refers to establishing the effectiveness of a system in light of its intended purposes Verification indicates how correct a given system can solve its proposed tasks (Watson) Verification indicates how correct a given system can solve its proposed tasks (Watson) Retrieval accuracy is indicated by the result given by the system when the target case is part of the case collection. Retrieval accuracy is indicated by the result given by the system when the target case is part of the case collection. validation & verification (i)

18. Copyright Rosina Weber validation & verification (ii) Retrieval consistency: the same retrieval when executed the second time must retrieve exactly the same cases (e.g., with the same similarity if k- NN is used) Retrieval consistency: the same retrieval when executed the second time must retrieve exactly the same cases (e.g., with the same similarity if k- NN is used) Case Duplication: when two distinct cases receive the same value for similarity in relation to a given target case. Case Duplication: when two distinct cases receive the same value for similarity in relation to a given target case. When the same value is attributed to different cases the user or the system has to decide which one to use by evaluating the value for each attribute. The same measure of similarity does not mean the cases necessarily teach the same lessons. When the same value is attributed to different cases the user or the system has to decide which one to use by evaluating the value for each attribute. The same measure of similarity does not mean the cases necessarily teach the same lessons.

19. Copyright Rosina Weber Case Coverage is checked for the evenly distribution of cases when they are manipulated and not actual experiences that are collected as they happen. Case Coverage is checked for the evenly distribution of cases when they are manipulated and not actual experiences that are collected as they happen. Efficiency verification : comparison to alternative methods, empirical evaluations Efficiency verification : comparison to alternative methods, empirical evaluations Retrieval time Retrieval time Retrieval sorting Retrieval sorting Case base consistency can be indicated by retrievals resulting cases with gradual values of similarity. A retrieval that no case has a high value of similarity or too many cases have the same value suggests inconsistency in the case base Case base consistency can be indicated by retrievals resulting cases with gradual values of similarity. A retrieval that no case has a high value of similarity or too many cases have the same value suggests inconsistency in the case base validation & verification (iii)

20. Copyright Rosina WeberMaintenance if the reasoner learns, the maintenance is more elaborate if the reasoner learns, the maintenance is more elaborate statistics of case usage statistics of case usage perform validation tests continuously perform validation tests continuously special issue on case-based maintenance special issue on case-based maintenance Neural networks and other soft computing methods have been proposed Neural networks and other soft computing methods have been proposed methods for distributed case bases methods for distributed case bases D. B. Leake, B. Smyth, D. C. Wilson, Q. Yang, “Special issue on maintaining case- based reasoning systems,” Computational Intelligence, 17(2), pp.193-195, 2001. D. B. Leake, B. Smyth, D. C. Wilson, Q. Yang, “Special issue on maintaining case- based reasoning systems,” Computational Intelligence, 17(2), pp.193-195, 2001.

21. Copyright Rosina Weber Determine the domain and scope Determine the domain and scope Consider reusing existing ontologies Consider reusing existing ontologies Enumerate important terms in the ontology Enumerate important terms in the ontology Define the classes and the class hierarchy Define the classes and the class hierarchy Define the attributes of classes (slots) Define the attributes of classes (slots) Define the facets of the slots Define the facets of the slots Create instances Create instances from Noy & McGuinness Ontologies Development Life Cycle Ontologies Development Life Cycle

22. Copyright Rosina Weber Ontologies Development Process Ontologies Development Process ontology development is an iterative process ontology development is an iterative process determine scope consider reuse enumerate terms define classes consider reuse enumerate terms define classes define properties create instances define classes define properties define constraints create instances define classes consider reuse define properties define constraints create instances from Noy & McGuinness

23. Uncertainty

24. Copyright Rosina Weber Uncertainty in Information Forms of ignorance: incompleteness: incompleteness: value of a variable is missing value of a variable is missing imprecision: imprecision: value of a variable is given but not with the precision required value of a variable is given but not with the precision required ambiguity: ambiguity: more than one possible value of a variable more than one possible value of a variable uncertainty: uncertainty: the value of a variable might be wrong the value of a variable might be wrong

25. Copyright Rosina Weber Uncertainty in Elements Fuzziness or vagueness Fuzziness or vagueness refers to ill-defined bounds to describe an element. refers to ill-defined bounds to describe an element. Randomness Randomness refers to the certainty of whether a given element belongs or not to a well-defined set. refers to the certainty of whether a given element belongs or not to a well-defined set. Probability Probability quantifies the chance that an event might occur or the belief that a proposition is true. quantifies the chance that an event might occur or the belief that a proposition is true.

26. Copyright Rosina Weber Uncertain Problems uncertain knowledge uncertain knowledge uncertain reasoning uncertain reasoning When trying to solve a problem: input --> reasoning --> output What’s the impact on the output when uncertainty of the input is not treated?

27. Copyright Rosina Weber Methods for the Treatment of Uncertainty

28. Copyright Rosina Weber Fuzzy Set Theory Fuzzy Sets Fuzzy Sets Fuzzy Logic Fuzzy Logic Extension principle Extension principle

29. Copyright Rosina Weber Set of American cities in the East Philadelphia LA New York q=1 q=0 q=1 q=? q=0 Fuzzy Sets Wilmington Set of big American cities New York DE Wilmington Philadelphia LA

30. Copyright Rosina Weber Fuzzy Sets Degree of membership ( q ) Degree of membership ( q ) Extension Principle Extension Principle L.Zadeh, A.Kandel, R. Yager, G.Klir, H. Zimmermann, B. Kosko L.Zadeh, A.Kandel, R. Yager, G.Klir, H. Zimmermann, B. Kosko

31. Copyright Rosina Weber Aggregations soundimageWM 50 100050 40 FI: fuzzy integral WM: weighted mean FI 0 40

32. Copyright Rosina Weber Theory of Probability  Measures degree of belief  80% degree of belief is a fairly strong expectation  What does x% chance means?  Frequentist approach  x% means that x is expected for each 100 times  Probability summarizes the uncertainty that comes from ignorance  Must follow statistical principles, for example additivity and complementarity

33. Copyright Rosina Weber Certainty Factors- CF ignores complementarity ignores complementarity Belief Belief Disbelief Disbelief Positive CF=> belief > disbelief Positive CF=> belief > disbelief Negative CF=> disbelief> belief Negative CF=> disbelief> belief Zero CF-> belief = disbelief and they both can be zero Zero CF-> belief = disbelief and they both can be zero

34. Copyright Rosina Weber Uncertainty in knowledge-based systems

35. Copyright Rosina Weber knowledge base (frames, uncertain rules) knowledge base (frames, uncertain rules) explanation general knowledge user I n t e r f a c e user I n t e r f a c e expert problem expert solution uncertain inference engine working memory (uncertain information) working memory (uncertain information) knowledge acquisition knowledge acquisition uncertainty in ES (i)

36. Copyright Rosina Weber uncertainty in ES (ii) Fuzzy rules If the students are tall then recruit them to the basketball team Fuzzy Logic Fuzzy implications define truth values of propositions; generalized modus ponens, fuzzy modus ponens Rules with Certainty Factors IF liquidity is very high (0.6) AND benefit is moderate (0.8) THEN ask for discount

37. Copyright Rosina Weber uncertainty in ES (iii) Fuzzy information The patient is middle-aged, exercises regularly, and is slightly overweight Fuzzy inference engine Grades of membership in inference engines using fuzzy implications

38. Copyright Rosina Weber Rules can have Certainty Factors IF liquidity is very high AND benefit is moderate (0.8) THEN purchase cash

39. Copyright Rosina Weber solution Retain Fuzzy Retrieval Revise Reuse proposed input problem with uncertain information confirmed solution initial solutions case base case adaptation with fuzzy rules uncertainty in CBR (i) solution CBR assumptions: problems recur similar problems have similar solutions situation assessment

40. Copyright Rosina Weber uncertainty in CBR (ii) prediction: prediction: the set of most similar and look for the most likely to be the solution the set of most similar and look for the most likely to be the solution summary or combination of the solutions with measures of central tendency or analogous summary or combination of the solutions with measures of central tendency or analogous

41. Copyright Rosina Weber uncertainty in CBR (iii) retrieval target casecandidate case weight attr 1a1ta1cw 1 attr 2a2ta2cw 2 attr 3a3ta3cw 3 nearest neighbor algorithms implement synthetic evaluation of similarity benefit of fuzzy methods is the relaxation of the additivity axiom

42. Copyright Rosina Weber Uncertainty in Ontologies Ignorance in classes, objects, facets incompleteness incompleteness imprecision imprecision ambiguity ambiguity uncertainty uncertainty Elements and facts (parameters) fuzziness or vagueness fuzziness or vagueness randomness randomness probability probability Uncertain reasoning: methods, functions, axioms

43. probability vs. possibility

44. Copyright Rosina Weber Probability Distribution Frequentist approach Observing John eating breakfast 100 times freqeggs 652 351 00,3,4,5,… However, it would still be very easy for him to eat 3 or 0 eggs, so even though these would have very low probability, they would have very high possibility.

45. Copyright Rosina Weber Possibility Distribution How "easy" it is for John to eat eggs for breakfast PD= 2/0.9 + 3/0.9 + 4/0.75+5/0.6 +6/0.3+ 7/0.05

46. Copyright Rosina Weber possibilityprobabilitydistribution 1 egg0.80.35 2 eggs0.90.65 3 eggs0.90.0 4 eggs0.750.0 5 eggs0.60.0 6 eggs0.30.0 7 eggs0.050.0 Methods based on probability are based on classic binary logic and must meet axioms such as additivity Methods based on Fussy Set Theory have weaker axioms

47. Copyright Rosina Weber probability vs. possibility What’s the difference in a real application? Probability is easier to acquire Probability is easier to acquire Ease of happening requires elicitation Ease of happening requires elicitation Possibility seems more amenable to specific and limited domains where the ‘ease’ will be available Possibility seems more amenable to specific and limited domains where the ‘ease’ will be available More general problems should be more difficult to validate with possibility, thus more suited to be dealt with probability More general problems should be more difficult to validate with possibility, thus more suited to be dealt with probability

48. Some solutions for some problems

49. Copyright Rosina Weber What are the problems and solutions? Incompleteness: value of a variable is missing In CBR, having all values is a requirement; In ES, a rule cannot be triggered; In NN, an example will not be trained; Solutions: substitute the missing value for a degree of belief or truth (probability); create rules that presents a direction for incomplete values based on domain knowledge; Replace incomplete variables with fuzzy sets;

50. Copyright Rosina Weber Imprecision: value of a variable is given but not with the precision required (e.g., grades) Why does it happen? originally imprecise (age in yrs) originally imprecise (age in yrs) generated by imprecise method generated by imprecise method need to work at an abstract level need to work at an abstract level summarized summarized Solutions: replace values for intervals; define degrees of belief/degrees of truth; reason at an abstract level; measure the imprecision and convey its effects to the final result;

51. Copyright Rosina Weber Ambiguity: more than one possible value of a variable; possible values are known Solutions: Who can disambiguate? Depends on the source of clarification. In listening, text understanding, information extraction degree of belief, truth; ontologies choose which is the most likely meaning (with uncertainty) test the meaning in similar natural language sources (e.g., text, people);

52. Copyright Rosina Weber Uncertainty: the value of a variable might be wrong prediction; prediction; during reasoning, partial values; during reasoning, partial values; when you must defuzzify or eliminate the degree of belief; when you must defuzzify or eliminate the degree of belief; Solutions: reduce the uncertainty, where does it come from? typicality of a set as an alternative to measures of central tendency; degree of belief; fuzzy implications for possible values, possibility distribution;