1 CHAPTER 11 Knowledge Acquisition and Validation Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

2 Knowledge Acquisition and Validation Knowledge Engineering Knowledge acquisition, representation, validation, inferencing, explanation and maintenance – Involves the cooperation of human experts – Synergistic effect Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

3 Knowledge Engineering Process Activities n Knowledge Acquisition n Knowledge Validation n Knowledge Representation n Inferencing n Explanation and Justification Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

4 Knowledge Engineering Process (Figure 11.1) Knowledge validation (test cases) Knowledge Representation Knowledge Acquisition Encoding Inferencing Sources of knowledge (experts, others) Explanation justification Knowledge base Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

5 Scope of Knowledge n Knowledge acquisition is the extraction of knowledge from sources of expertise and its transfer to the knowledge base and sometimes to the inference engine n Knowledge is a collection of specialized facts, procedures and judgment rules Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

6 Knowledge Sources n Documented (books, manuals, etc.) n Undocumented (in people's minds) – From people, from machines n Knowledge Acquisition from Databases n Knowledge Acquisition Via the Internet Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

7 Knowledge Levels n Shallow knowledge (surface) n Deep knowledge n Can implement a computerized representation that is deeper than shallow knowledge n Special knowledge representation methods (semantic networks and frames) to allow the implementation of deeper-level reasoning (abstraction and analogy): important expert activity n Represent objects and processes of the domain of expertise at this level n Relationships among objects are important Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

8 Major Categories of Knowledge n Declarative Knowledge n Procedural Knowledge n Metaknowledge Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

9 Declarative Knowledge Descriptive Representation of Knowledge n Expressed in a factual statement n Shallow n Important in the initial stage of knowledge acquisition Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

10 Procedural Knowledge n Considers the manner in which things work under different sets of circumstances – Includes step-by-step sequences and how-to types of instructions – May also include explanations – Involves automatic response to stimuli – May tell how to use declarative knowledge and how to make inferences Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

11 n Descriptive knowledge relates to a specific object. Includes information about the meaning, roles, environment, resources, activities, associations and outcomes of the object n Procedural knowledge relates to the procedures employed in the problem-solving process Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

12 Metaknowledge Knowledge about Knowledge In ES, Metaknowledge refers to knowledge about the operation of knowledge-based systems Its reasoning capabilities Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

13 Required Knowledge Engineer Skills n Computer skills n Tolerance and ambivalence n Effective communication abilities n Broad educational background n Advanced, socially sophisticated verbal skills n Fast-learning capabilities (of different domains) n Must understand organizations and individuals n Wide experience in knowledge engineering n Intelligence n Empathy and patience n Persistence n Logical thinking n Versatility and inventiveness n Self-confidence Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

14 Knowledge Acquisition Methods: An Overview n Manual n Semiautomatic (Expert-driven) n Automatic (Computer Aided - Induction driven) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

15 Manual Methods - Structured Around Interviews n Process (Figure 11.4) n Interviewing n Tracking the Reasoning Process n Observing n Manual methods: slow, expensive and sometimes inaccurate Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

16 Manual Methods of Knowledge Acquisition Elicitation Knowledge base Documented knowledge Experts Coding Knowledge engineer Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

17 Expert-Driven Knowledge Acquisition Knowledge base Knowledge engineer Expert Coding Computer-aided (interactive) interviewing Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

18 Induction-Driven Knowledge Acquisition Knowledge base Case histories and examples Induction system Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

19 Interviews n Most Common Knowledge Acquisition: Face-to-face interviews n Interview Types – Unstructured (informal) – Semi-structured – Structured Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

20 Unstructured Interviews n Most Common Variations – Talkthrough – Teachthrough – Readthrough Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

21 n The knowledge engineer slowly learns about the problem n Then can build a representation of the knowledge n Knowledge acquisition involves – Uncovering important problem attributes – Making explicit the expert’s thought process Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

22 Structured Interviews n Systematic goal-oriented process n Forces an organized communication between the knowledge engineer and the expert n Procedural Issues in Structuring an Interview n Interpersonal communication and analytical skills are important Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

23 Recommendation Before a knowledge engineer interviews the expert(s) 1. Interview a less knowledgeable (minor) expert – Helps the knowledge engineer Learn about the problem Learn its significance Learn about the expert(s) Learn who the users will be Understand the basic terminology Identify readable sources 2. Next read about the problem 3. Then, interview the expert(s) (much more effectively) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

24 Tracking Methods n Techniques that attempt to track the reasoning process of an expert n From cognitive psychology n Most common formal method: Protocol Analysis Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

25 Protocol Analysis n Protocol : a record or documentation of the expert's step-by-step information processing and decision-making behavior n The expert performs a real task and verbalizes his/her thought process (think aloud) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

26 Expert-driven Methods n Knowledge Engineers Typically – Lack Knowledge About the Domain – Are Expensive – May Have Problems Communicating With Experts n Knowledge Acquisition May be Slow, Expensive and Unreliable n Can Experts Be Their Own Knowledge Engineers? Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

27 Approaches to Expert-Driven Systems n Manual n Computer-Aided (Semiautomatic) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

28 Computer-aided Approaches n To reduce or eliminate the potential problems – REFINER+ - case-based system – TIGON - to detect and diagnose faults in a gas turbine engine n Other – Visual modeling techniques – New machine learning methods to induce decision trees and rules – Tools based on repertory grid analysis Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

29 Repertory Grid Analysis (RGA) n Techniques, derived from psychology n Use the classification interview n Fairly structured n Primary Method: Repertory Grid Analysis (RGA) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

30 The Grid n Based on Kelly 's model of human thinking: Personal Construct Theory (PCT) n Each person is a "personal scientist" seeking to predict and control events by – Forming Theories – Testing Hypotheses – Analyzing Results of Experiments n Knowledge and perceptions about the world (a domain or problem) are classified and categorized by each individual as a personal, perceptual model n Each individual anticipates and then acts Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

31 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

32 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

33 RGA in Expert Systems - Tools n AQUINAS – Including the Expertise Transfer System (ETS) n KRITON n PCGRID (PC-based) n WebGrid n Circumgrids Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

34 Knowledge Engineer Support n Knowledge Acquisition Aids n Special Languages n Editors and Interfaces n Explanation Facility n Revision of the Knowledge Base n Pictorial Knowledge Acquisition (PIKA) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

35 n Integrated Knowledge Acquisition Aids – PROTÉGÉ-II – KSM – ACQUIRE – KADS (Knowledge Acquisition and Documentation System) n Front-end Tools – Knowledge Analysis Tool (KAT) – NEXTRA (in Nexpert Object) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

36 Machine Learning n Knowledge Discovery and Data Mining n Include Methods for Reading Documents and Inducing Knowledge (Rules) n Other Knowledge Sources (Databases) n Tools – KATE-Induction – CN-2 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

37 Automated Rule Induction n Induction: Process of Reasoning from Specific to General n In ES: Rules Generated by a Computer Program from Cases n Interactive Induction Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

38 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

39 Case-based Reasoning (CBR) n For Building ES by Accessing Problem-solving Experiences for Inferring Solutions for Solving Future Problems n Cases and Resolutions Constitute a Knowledge Base Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

40 Neural Computing n Fairly Narrow Domains with Pattern Recognition n Requires a Large Volume of Historical Cases Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

41 Intelligent Agents for Knowledge Acquisition Led to n KQML (Knowledge Query and Manipulation Language) for Knowledge Sharing n KIF, Knowledge Interchange Format (Among Disparate Programs) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

42 Validation and Verification of the Knowledge Base n Quality Control – Evaluation – Validation – Verification Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

43 n Evaluation – Assess an expert system's overall value – Analyze whether the system would be usable, efficient and cost-effective n Validation – Deals with the performance of the system (compared to the expert's) – Was the “right” system built (acceptable level of accuracy?) n Verification – Was the system built "right"? – Was the system correctly implemented to specifications? Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

44 Dynamic Activities n Repeated each prototype update n For the Knowledge Base – Must have the right knowledge base – Must be constructed properly (verification) n Activities and Concepts In Performing These Quality Control Tasks Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

45 To Validate an ES n Test 1. The extent to which the system and the expert decisions agree 2. The inputs and processes used by an expert compared to the machine 3. The difference between expert and novice decisions (Sturman and Milkovich [1995]) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

46 Analyzing, Coding, Documenting, and Diagramming Method of Acquisition and Representation 1. Transcription 2. Phrase Indexing 3. Knowledge Coding 4. Documentation (Wolfram et al. [1987]) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

47 Knowledge Diagramming n Graphical, hierarchical, top-down description of the knowledge that describes facts and reasoning strategies in ES n Types – Objects – Events – Performance – Metaknowledge n Describes the linkages and interactions among knowledge types n Supports the analysis and planning of subsequent acquisitions n Called conceptual graphs (CG) n Useful in analyzing acquired knowledge Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

48 Numeric and Documented Knowledge Acquisition n Acquisition of Numeric Knowledge – Special approach needed to capture numeric knowledge n Acquisition of Documented Knowledge – Major Advantage: No Expert – To Handle a Large or Complex Amount of Information – New Field: New Methods That Interpret Meaning to Determine Rules Other Knowledge Forms (Frames for Case-Based Reasoning) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

49 The Internet/Intranet for Knowledge Acquisition n Electronic Interviewing n Experts can Validate and Maintain Knowledge Bases n Documented Knowledge can be accessed n The Problem: Identifying relevant knowledge (intelligent agents) n Many Web Search Engines have intelligent agents n Data Fusion Agent for multiple Web searches and organizing n Automated Collaborative Filtering (ACF) statistically matches peoples’ evaluations of a set of objects Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

50 Induction Table Example n Induction tables (knowledge maps) focus the knowledge acquisition process n Choosing a hospital clinic facility site Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

51 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ