1. Chapter 6 Supplement Knowledge Engineering and Acquisition Chapter 6 Supplement

2. Knowledge Acquisition Knowledge acquisition is the extraction of knowledge from sources of expertise and its transfer to the knowledge base and sometimes to the inference engine

3. What are some of the Difficulties in Knowledge Acquisition Expressing the knowledge: Human knowledge exists in a compiled format. A human doesn’t remember all the intermediate steps used to in transferring and processing knowledge – representation mismatch Number of participants Structuring the knowledge: We must elicit not only the knowledge but also its structure; rules “Knowers” lack time and unwilling to help Testing and refining knowledge is hard Collect knowledge from one source but relevant knowledge is dispersed Important knowledge may be mixed up with irrelevant information Incomplete knowledge (use one source only) “Knowers” may change their behavior when observed Problematic interpersonal factors

4. Knowledge Engineering Process Activities Knowledge Acquisition Acquisition of knowledge from human experts, books, documents, or computer files Knowledge Validation Knowledge is validated and verified (using test cases) until the quality is acceptable Knowledge Representation Organized knowledge; creation of a knowledge map and the encoding of knowledge into a knowledge base Inferencing Design of software to enable the software to make inferences based on the knowledge and the specifics of the a problem Explanation and Justification The design and programming of an explanation capability. Why is this piece of information needed? How was a certain conclusion derived.

5. Knowledge Engineering Process Knowledge validation (test cases) Knowledge Representation Knowledge Acquisition Encoding Inferencing Sources of knowledge (experts, others) Explanation justification Knowledge base

6. Knowledge Sources Documented (books, manuals, etc.) Undocumented (in people's minds) From people, from machines Knowledge Acquisition from Databases Knowledge Acquisition Via the Internet

7. Knowledge Acquisition Methods: An Overview Manual :the knowledge engineer interacts directly with the experts Interviews, tracking the reasoning process (protocol analysis), observing, brainstorming, conceptual graphs and models Semiautomatic (Expert-driven): the expert encodes his or her expertise directly into the computer system or the developer uses technology to facilitate the knowledge acquistion Expert’s self reports, computer aided approaches (visual modeling); graphical development environment where the initial knowledge domain can be modeled and manipulated (decision trees based on business process logic) ex. REFINER+ patient manager Automatic (Computer Aided - Induction driven) Minimize or eliminate the role of the KE and/or the expert inference engines extract the knowledge from a set of examples

8. Manual Methods of Knowledge Acquisition Elicitation Knowledge base Documented knowledge Experts Coding Knowledge engineer

9. Expert-Driven Knowledge Acquisition Knowledge base Knowledge engineer Expert Coding Computer-aided (interactive) interviewing

10. Induction-Driven Knowledge Acquisition Knowledge base Case histories and examples Induction system

11. Manual Acquisition Techniques Interviewing: two common types are unstructured (conversational) and structured (interrogation/using a script) Verbal Protocol Analysis: Most of the information necessary to model knowledge is found in the cognitive process the knower uses to solve a problem/do a task Document the step-by-step information processing and decision making behavior by the knower Concurrent: Think aloud or verbalize thoughts while doing task Repertory Grid Method: Maybe manual or computerized

12. Expert Driven/Computer Aided Reparatory Grid Analysis May also be employed by the KE Developed by Kelly (1955) who conceived humans as ”personal scientist” each with their own model of the world. the expert compares successive groups of three objects and tells why two differ from the third Also used to infer similarities in construct beliefs held by multiple experts Knowledge and perceptions about the world are classified and categorized by each individual as a personal, perceptual model.

13. Machine learning/Automated Rule Induction Training set: example of a problem for which the outcome is known After given enough examples, the rule induction system can create rules that fit the example cases. The rules can be used to assess new cases for which the outcome is not known. For Example: Loan Officer’s tasks: Requests for loans include information about the applicants such as income, assets, age and number of dependents

15. From this case, it is easy to derive the following three rules: If Income is $70,000 or more approve the loan If income is $30,000 or more, age is at least 40, assets are above $249,000 and there are no dependents approve the loan If income is between $30,000 and $50,000 and assets are at least $100,000, approve the loan

16. Multisource Knowledge Acquisition It is likely that multiple sources will be needed to fully acquire the knowledge for a problem and conflicting views and opinions often arise. Brainstorming/Electronic Brainstorming Goal is to come up with creative solutions. Idea generation and evaluation Consensus Decision NGT Delphi Method Concept Mapping Blackboarding

17. Validation and Verification of the Knowledge Base Quality Control Evaluation Validation Verification

18. Evaluation Assess an expert system's overall value Analyze whether the system would be usable, efficient and cost-effective Validation Deals with the performance of the system (compared to the expert's) Was the “right” system built (acceptable level of accuracy?) Verification Was the system built "right"? Was the system correctly implemented to specifications?

19. To Validate an ES Test 1. The extent to which the system and the expert decisions agree 2. The inputs and processes used by an expert compared to the machine 3. The difference between expert and novice decisions

20. Some validation measures Accuracy Adaptability Adequacy Breadth Depth Face Validity Generality Precision Realism Reliability Robustness Usefulness