1. Closing Some Loose Ends Sources: David W. Aha My own Thomas H. Davenport, Laurence Prusak, 1998

2. Classifiers

3. Case-based reasoning (CBR) classifier Induction of decision trees (IDT) CBR+IDT classifier Others (e.g., covered in the Data Mining course):  Support Vector Machines  Linear regression  Neural networks  … So which one is best?

4. No Free Lunch Theorem Each of these classifiers have a bias To explain the bias, let us examine a situation where instances (or cases) are pairs of numeric features and a binary classification problem: ((x,y),class) Let us draw the space: CBR, K-d trees (K=2), Support vector machines Let us construct examples where each of these classifiers works best  How does the other classifiers work on these examples? Formulation of the no free lunch theorem

5. Knowledge Management

6. The Beginning: The Apollo 13 Situation http://www.youtube.com/watch?v=nEl0NsYn1fU http://www.youtube.com/watch?v=nEl0NsYn1fU The oxygen tanks had originally been designed to run off the 28 volt DC The tanks were redesigned to also run off the 65 volt DC

7. The Changing Game The New Economics ManufacturingService TangibleIntangible ConsumableInconsumable StructuralIntellectual Tobin’s Q ratio company’s stock market value / value of its physical assets company’s stock market value / value of its physical assets Is increasing dramatically. What does this mean? Increasing importance of intellectual capital in the United States (Barr & Magaldi, 1996)

8. Knowledge Management (KM) An increasingly important new business movement that promotes the creation, sharing, & leveraging of knowledge within an organization to maximize business results. Effective tools to capture, leverage & reuse knowledge Effective tools to capture, leverage & reuse knowledge Technology Develop a culture for knowledge sharing Develop a culture for knowledge sharing Organizational Dynamics Needs Financial constraints Loss of organizational knowledge Financial constraints Loss of organizational knowledge Needs Problems:

9. Knowledge Management: Issues Technical and Business Expertise:  Proficiencies  Know-How  Skills Work Practice Execution:  Processes  Methodologies  Practices  Lessons learned

11. Why Knowledge Management? Leverages Core Business Competence Accelerates Innovation (Time to Market) Improves Cycle Times (Market to Collection) Improves Decision Making Strengthens Organizational Commitment Builds sustainable differentiation

12. CBR: The Knowledge Management Plunge “Case-based reasoning programs have been shown to bring about marked improvements in customer service.” - Thomas H. Davenport, Laurence Prusak, 1998 - Working Knowledge: How Organizations Manage What They Know “Case-based reasoning programs have been shown to bring about marked improvements in customer service.” - Thomas H. Davenport, Laurence Prusak, 1998 - Working Knowledge: How Organizations Manage What They Know KM CBRWorks eGain eService Enterprise (E3)

13. KM Project Domains: CBR Applicable? (KM World, 1/99, Dan Holtshouse, Xerox) 1. Sharing knowledge and best practices 2. Instilling responsibility for knowledge sharing 3. Capturing and reusing past experiences 4. Embedding knowledge (products/services/processes) 5. Producing knowledge as a product 6. Driving knowledge generation for innovation 7. Mapping networks of experts 8. Building/mining customer knowledge bases 9. Understanding/mining customer knowledge bases 10. Leveraging intellectual assets. KM Domains/TasksCBR Applicable? Yes No Yes No Yes No Yes No

14. 1999 AAAI KM/CBR Workshop ~45 attendees: Siemens, Schlumberger, Motorola, NEC, British Airways, General Motors, Boeing, Ford Motor Company, World Bank Goals: 1. Explain KM issues to CBR researchers 2. Report on recent CBR approaches for KM tasks 3. Share cautions, knowledge, & experiences Goals: 1. Explain KM issues to CBR researchers 2. Report on recent CBR approaches for KM tasks 3. Share cautions, knowledge, & experiences Some observations: 1. Embedded/integrated in knowledge processes 2. Benefits of semi-structured case representations 3. Interactive (“conversational”) systems Some observations: 1. Embedded/integrated in knowledge processes 2. Benefits of semi-structured case representations 3. Interactive (“conversational”) systems

15. Limitations of CBR for KM (from the 1999 AAAI KM/CBR Workshop) 1. Main limitation is time and effort? (Wess/Haley) 2. Limitations from working with simple representations (Haley) –Becoming less problematic (e.g., with development of textual CBR) 2. Limitations from working with simple representations (Haley) –Becoming less problematic (e.g., with development of textual CBR) 3. Rule-based integrations –Suffer from old problems of rule acquisition –But KM problem-solving techniques are combating this (Studer) 3. Rule-based integrations –Suffer from old problems of rule acquisition –But KM problem-solving techniques are combating this (Studer) 4. More intuitive case authoring capabilities 5. Tools for working with heterogeneous data sources

16. Panel: Lessons & Suggested Directions CBR Roles: –Accumulate, extend, preserve, distribute, reuse corporate knowledge –Extracting tacit knowledge –Customer relationship management CBR Roles: –Accumulate, extend, preserve, distribute, reuse corporate knowledge –Extracting tacit knowledge –Customer relationship management Lessons & Observations: –Integrate CBR with KM tasks & task models –Integrate case retrieval with presentation with tools/workplaces –Integrate case construction/indexing with work product development –Need more advanced (automated) case authoring tools –Must consider effects on user groups, time, organizational impact –CBR not a complete KM solution Lessons & Observations: –Integrate CBR with KM tasks & task models –Integrate case retrieval with presentation with tools/workplaces –Integrate case construction/indexing with work product development –Need more advanced (automated) case authoring tools –Must consider effects on user groups, time, organizational impact –CBR not a complete KM solution

17. Experience Management vs CBR Experience Management CBR (Organization) (IDSS) 2. Reuse 3. Revise 4. Retain Case Library 1. Retrieve Background Knowledge Experience base Reuse- related knowledge Problem acquisition Experience evaluation and retrieval Experience adaptation Experience presentation Complex problem solving Development and Management Methodologies BOOK

18. Relating KM with AI AI Knowledge-Based Systems Human Factors KM Business Processing CBR

19. Distinguishing KM from Data Mining KDD Focus: Large databases Autonomous pattern recognition Knowledge Discovery from Databases Process: Database Acquisition Data Warehousing Data Cleansing Data Mining Data Maintenance KM Focus: Capturing organizational dynamics processes Interaction (i.e., decision support)

20. Process-Oriented CBR Most KM tasks are performed in the context of a well- defined (e.g., business) process, and any techniques designed to support KM must be embedded in this process KM examples (many): Enterprise resource planning (O’Leary) Project process (Maurer & Holz) KM examples (many): Enterprise resource planning (O’Leary) Project process (Maurer & Holz) CBR examples (few): Leake et al.: Feasibility assessment in design process Moussavi, Shimazu: Cases represent processes Reddy & Munoz-Avila: Project Planning CBR examples (few): Leake et al.: Feasibility assessment in design process Moussavi, Shimazu: Cases represent processes Reddy & Munoz-Avila: Project Planning

21. Motivation for Design Project Embedding CBR into existing tools has been shown to be an effective way to insert CBR into KM processes  We saw it this year in a number of projects:  Help-desk for LTS  Recommender for university events  Companies processes We discuss two applications  They have a similar flavor to most of the design projects

22. Two Examples

23. EXTERNAL MONITORING Alerts Spiders Workflow Scheduling Collaboration Suspenses Records Management Document Management E-mail OA tools Library catalog Online databases E-journals How-to guides Document Delivery Service Bulletin boards Buckets Profiles MIS INFORMATION SOURCES WORKSPACE PERSONAL PORTAL AFRL Proposed KM Environment (multi?) impersonal

24. Personalization Semantic Web Ontologies DS1 DS2 DS3 Distributed data sources Assistant Agent Case Repository Causal Model Current Problem User Ontologies Personal Portal/ Workspace Information Sources

25. Individualized Portal Information Domains Data Systems Virtual Library Buckets Finance Personnel Executive Information System

26. Out-of-Family Disposition (OOFD) Process NASA-Kennedy Space Center: Shuttle Processing Directorate KM expertise CBR expertise Topic: Performing project tasks outside range of expertise Lack of task familiarity Motivations: Downsizing, employee loss, technology pace Resources: Interim problem reports Standardized text documents for reporting problems/solutions Given: 12 of these reports Topic: Performing project tasks outside range of expertise Lack of task familiarity Motivations: Downsizing, employee loss, technology pace Resources: Interim problem reports Standardized text documents for reporting problems/solutions Given: 12 of these reports Pre-flight, launch, landing, recovery Prof. I. Becerra-Fernandez

27. Example KM Aplication: SMART KM Portal SMART: Science Mission Assistant & Research Tool Categorization: An interactive, web-based tool suite Purpose: Reduce time/cost required to define new science initiatives SMART: Science Mission Assistant & Research Tool Categorization: An interactive, web-based tool suite Purpose: Reduce time/cost required to define new science initiatives Uncertainty

28. SMART is Architected as a Web Portal SMART User Web Browser http://smart.gsfc.nasa.gov SMART Intelligent Data Prospector Intelligent Data Prospector Find data sets Intelligent Resource Prospector Intelligent Resource Prospector Find an observatory Intelligent Mission Design Asst Intelligent Mission Design Asst Design a science mission http://smart.gsfc.nasa.gov/irp/ Browse Observatory Knowledge Base Map Tree Observatory Lists Search Observatory Knowledge Base Word/Phrase Search Interactive Dialog Discussions Experts Map Tree Observatory Lists Interactive Dialog Discussions Experts SMART Intelligent Resource Prospector http://smart.gsfc.nasa.gov/imda/ Browse Mission Knowledge Base Map Tree Mission Lists Search Mission Knowledge Base Word/Phrase Search Interactive Dialog Discussions Experts Map Tree Mission Lists Interactive Dialog Discussions Experts Design a Mission SMART Intelligent Mission Design Asst SMART Concept Map Viewer: Observatories SMART Hierarchical Directory Viewer SMART Database Views SMART Conversational CBR Question/Response Interface SMART Collaborative Discussions Interface SMART IMDA Design a Mission Create/Edit a Mission Create/Edit a Mission Validate Design Power Design Advisor Thermal Design Advisor Communications Design Advisor … Power Design Advisor Thermal Design Advisor Communications Design Advisor Invoke Design Validation Agent (applet) (server DB access) (applet) (KM tool service) (expert systems)

29. Searching for Missions Using CCBR SMART Conversational Mission Search Engine Describe what you are looking for: “I’m looking for astronomy missions in low-Earth orbit.” Ranked questions: Score Answer Name Title “X-ray”Q17What portion of the spectrum is observed? 60Q7What launch vehicle? 50Q32What mission phase? 20Q23Low or high inclination orbit? 10Q41Cryogenically-cooled instrument? Ranked cases: ScoreNameTitle 90XTEX-Ray Timing Explorer 90AXAFChandra X-Ray Observatory 30GROGamma Ray Observatory 30EUVEExtreme Ultra-Violet Explorer Question: Q17 Title: What portion of the spectrum is observed? Description: What portion of the electro-magnetic spectrum are you interested in? Select your answer:  Visible light  Infra-red  Ultra-violet  Microwave  X-Ray  Radiowave  Gamma Ray

30. Lessons Learned Keywords: Philippines, evacuation, disaster relief, c 2, NEO, Fiery Vigil, etc. Observation: Assignment of air traffic controllers to augment host country controllers was critical to safe evacuation airfield operation. Discussion: The rapid build-up of military flight operations…overloaded the civilian host nation controllers. Military controllers maintained 24 hour operations.... Lesson Learned: Military air traffic controllers are required whenever a civilian airport is transformed into an intensive military operating area for contingency operations. Recommended Action: Ensure controllers and liaison teams are part of the evacuation package, and establish early liaison with host nation to coordinate an agreement on operational procedures. What How When

31. Joint Unified Lessons Learned System (JULLS) Database: 908 “scrubbed” lessons from the CINC’s (1991-) –Unclassified subset: 150 lessons (Armed Forces Staff College) 33 relate to NEOs Database: 908 “scrubbed” lessons from the CINC’s (1991-) –Unclassified subset: 150 lessons (Armed Forces Staff College) 33 relate to NEOs Lesson Format: 43 attributes –e.g., ID Number, submitting command, subject, date –Unified Joint Task List number –Content attributes: All in text format 6Keywords 6Observation 6Discussion 6Lesson learned 6Recommended action Lesson Format: 43 attributes –e.g., ID Number, submitting command, subject, date –Unified Joint Task List number –Content attributes: All in text format 6Keywords 6Observation 6Discussion 6Lesson learned 6Recommended action

32. Some Lessons Learned Centers/Systems Air Force o Air Force Automated Lessons Learned Capture and Retrieval System o Air Force Center for Knowledge Sharing Lessons Learned o Air Combat Command Center for Lessons Learned o Automated Lessons Learned Collection & Retrieval System Army o Center for Army Lessons Learned (CALL) o SARDA: Contracting Lessons Learned o US Army Europe - Lessons Learned System Coast Guard o Coast Guard Universal Lessons Learned Joint Forces o JCLL: Joint Center for Lessons Learned Marine Corps o Marine Corps Lessons Learned System Navy o NDC: Navy Doctrine Command Lessons Learned System o NAWCAD: Navy Combined Automated Lessons Learned o NAVFAC: Naval Facilities Engineering Command Lessons Learned System Government (non-military) o NASA Lessons Learned Information System o International Safety Lessons Learned Information System o NASA-Goddard: RECALL: Reusable Experience with CBR for Automating Lessons Learned) o NIST: Best Practices Hyperlinks o DoE: US Department of Energy Lessons Learned Other o Canadian Army Lessons Learned Centre o United Nations: UN Lessons Learned in Peacekeeping Operations Air Force o Air Force Automated Lessons Learned Capture and Retrieval System o Air Force Center for Knowledge Sharing Lessons Learned o Air Combat Command Center for Lessons Learned o Automated Lessons Learned Collection & Retrieval System Army o Center for Army Lessons Learned (CALL) o SARDA: Contracting Lessons Learned o US Army Europe - Lessons Learned System Coast Guard o Coast Guard Universal Lessons Learned Joint Forces o JCLL: Joint Center for Lessons Learned Marine Corps o Marine Corps Lessons Learned System Navy o NDC: Navy Doctrine Command Lessons Learned System o NAWCAD: Navy Combined Automated Lessons Learned o NAVFAC: Naval Facilities Engineering Command Lessons Learned System Government (non-military) o NASA Lessons Learned Information System o International Safety Lessons Learned Information System o NASA-Goddard: RECALL: Reusable Experience with CBR for Automating Lessons Learned) o NIST: Best Practices Hyperlinks o DoE: US Department of Energy Lessons Learned Other o Canadian Army Lessons Learned Centre o United Nations: UN Lessons Learned in Peacekeeping Operations

33. Lessons Learned Repositories: Functionality Center for Lessons Learned Center for Lessons Learned Documented Lessons Decision-Support Tool Decision-Support Tool Retrieval Tool Interface Retrieval Tool Interface Lessons Learned Repository Lessons Learned Repository Lessons Learned System Search queries Relevant lessons

34. Lessons Learned Systems: Unrealistic Assumptions The decision maker 1. has time to search for lessons, 2. knows where to search for lessons, 3. knows how to search for lessons, and 4. knows how to interpret retrieved lessons for their current decision-making context. The decision maker 1. has time to search for lessons, 2. knows where to search for lessons, 3. knows how to search for lessons, and 4. knows how to interpret retrieved lessons for their current decision-making context.

35. Decision Support Tool User Interface Active Lessons Learned Repositories Center for Lessons Learned Center for Lessons Learned Documented Lessons Retrieval Tool Interface Retrieval Tool Interface Lessons Learned Repository Lessons Learned Repository Lessons Learned System LL Agent: (CBR) Relevance Assessment Retrieval Interpretation Search queries Relevant lessons

36. Issues for Active Lessons Learned Documented Lessons LL Agent (CBR) User Case Library Case extraction Decision Support Tool Decision-Making Process 1. Case extraction methods 2. Case representation 3. Choice of decision support tool 4. Embedded LL agent behavior 1. Case extraction methods 2. Case representation 3. Choice of decision support tool 4. Embedded LL agent behavior

37. Lessons Learned: NEO Critiquing Example Compose an Intermediate Stage Base Tasks Scenario: 50 miles from ISB #1 30 miles from ISB #2 Commercial airfield Resources: Transport vehicles … Joint Air Command Military air traffic controller... Objects: 1. Planning tasks 2. Resources 3. Assignments 4. Task relations 5. Scenario Objects: 1. Planning tasks 2. Resources 3. Assignments 4. Task relations 5. Scenario Coordinate with local security forces Coordinate with airfield traffic controllers... Lesson Learned #13167-92740: Index: Coordinate w/ traffic controllers Lesson: If ISB is a commercial airfield, then assign military air traffic controllers to the evacuation package Lesson Learned #13167-92740: Index: Coordinate w/ traffic controllers Lesson: If ISB is a commercial airfield, then assign military air traffic controllers to the evacuation package Transport military air traffic controller to ISB

38. KM/CBR: Possible Future Directions 1. Applications –e-Commerce –Decision support systems Personalized –Knowledge discovery for databases? Yet KDD stresses need for many automated tasks 1. Applications –e-Commerce –Decision support systems Personalized –Knowledge discovery for databases? Yet KDD stresses need for many automated tasks 2. Multimodal systems –e.g., Shimazu: Audio tapes of customer dialogues –Information gathering –Learning assistants 2. Multimodal systems –e.g., Shimazu: Audio tapes of customer dialogues –Information gathering –Learning assistants 3. Process-focused emphases: –Retrieval, adaptation, and composition of processes 3. Process-focused emphases: –Retrieval, adaptation, and composition of processes