1. On the Technological, Human, and Managerial Issues in Sharing Organizational Lessons Intelligent Decision Aids Group Head: David W. Aha Navy Center for Applied Research in Artificial Intelligence Naval Research Laboratory Office of Naval Research and the University of Wyoming Rosina Weber, Leonard Breslow, Nabil Sandhu

2. Outline Context  lessons learned systems (LLS) Motivation: difficulties in sharing lessons with LLS  Technological  Human  Managerial Solution  TCBR approach for KM: Represents technological solutions to address technological, human and managerial problems Problems vs. Solutions How technology can address human and managerial issues 1 2 3 4 5 1 2 3 4 5

3. Context: lessons learned systems What:  Repository-type, process-oriented KM 1 2 3 4 5

4. Repository-based Knowledge Management Systems

5. Context: lessons learned systems What:  Lessons learned, organizational lessons 1 2 3 4 5

6. Organizational lessons are pieces of knowledge originated from experience in organizational processes Single experience. Each lesson should describe a positive or negative experience (suggesting to repeat or avoid a practice). Correctness. Lessons must be validated with respect to their processes and context. Applicability. Lessons must be applicable to a specific task or decision. Process-centric. When reused, the lesson must positively impact the targeted process.

7. Context: lessons learned systems Where  survey 1 2 3 4 5

8. non-government Construction Industry Institute Honeywell GM Bechtel Jacobs Company Lockheed Martin Energy Systems, Inc DynMcDermott Petroleum Operations Co. Xerox Bestbuy int’l Canadian Army Lessons Learned Centre military government US Air Force Army Coast Guard Joint Forces Marine Corps Navy non-military Department of Energy: SELLS NASA (Ames, Goddard) US int’l European Space Agency (ESA) Italian (Alenia) French (CNES) Japanese (NASDA) United Nations

9. Motivation: difficulties in sharing organizational lessons with LLS Technological  Development and performance Human  Lesson authors  Lesson users Managerial  Responsibilities wrt to submission and reuse of lessons 1 2 3 4 5

10. Technological problems  FLAIRS 2000:  Standalone dissemination outside context of reuse  AAAI 2000:  Low precision and recall in lesson dissemination  ISMIS 2000:  Textual representation of lessons  Experience is communicated in natural language  IJCAI 2001: monitored distribution  Requirement: Lessons in machine recognizable format 1 2 3 4 5

11. Technological problems  FLAIRS 2000:  Standalone dissemination outside context of reuse  AAAI 2000:  Low precision and recall in lesson dissemination  ISMIS 2000:  Textual representation of lessons  Experience is communicated in natural language  IJCAI 2001: monitored distribution  Requirement: Lessons in machine recognizable format 1 2 3 4 5 Technological: – Architecture of repository – Retrieval method – Format of lessons – Collection method

12. Human problems Lesson authors:  Lack of training/instructions: content and format Lesson validators  Hard to validate textual descriptions Lesson users  Have to access the repository in another context  Have to accept the potential benefit of lessons  Have the skills to search for lessons  Have to interpret textual lessons 1 2 3 4 5

13. Human problems Lesson authors:  Lack of training/instructions: content and format Lesson validators  Hard to validate textual descriptions Lesson users  Have to access the repository in another context  Have to accept the potential benefit of lessons  Have the skills to search for lessons  Have to interpret textual lessons 1 2 3 4 5 Human: – Collection method – Architecture of repository – Retrieval method – Format of lessons

14. Knowledge collection:  Determine, communicate and enforce standards for lesson collection and representation Knowledge validation:  Define structured format Knowledge reuse:  Embed knowledge in targeted processes  Monitor knowledge transfer  Oversee knowledge reuse Managerial problems 1 2 3 4 5

15. Knowledge collection:  Determine, communicate and enforce standards for lesson collection and representation Knowledge validation:  Define structured format Knowledge reuse:  Embed knowledge in targeted processes  Monitor knowledge transfer  Oversee knowledge reuse Managerial problems 1 2 3 4 5 Managerial : – Collection method – Architecture of repository – Retrieval method – Format of lessons

16. Issues in sharing lessons  Technological: –Architecture of repository –Retrieval method –Format of lessons –Collection method  Human: –Training/instructions to submit lessons –Standalone vs.embedded –Textual vs. structured format  Managerial: –Training/instructions to submit lessons –Standalone vs. embedded –Textual vs. structured format

17. Textual Case-Based Reasoning Framework for Knowledge Management TCBR framework for KM (GWCBR 2001) provides infra-structure to solve technological, human and managerial problems. case base artifacts in the format of distribution system elicitation tool domain specific ontology human users extraction tool text documents case base Case base of lessons lessons distributed in a distribution system Lesson elicitation tool domain specific taxonomy human users extraction tool textual lessons

18. Problems & Solutions (i) Standalone distribution: 1. Users may not be reminded of the repository, as they need to access a standalone tool to search for lessons 2. Users may not be convinced of the potential utility of lessons 3. Users may not have the time and skills to retrieve relevant lessons 4. Text databases have low levels of precision and recall Monitored distribution (Case-based): when and where 1. Lessons are brought to the attention of the user when and where they are needed 2. Users access the lesson rationale to evaluate its potential utility 3. No additional time, skills, or interpretation is required 4. Case retrieval of disambiguated knowledge increase recall and precision

19. Problems & Solutions (ii) Textual lessons: 1. Users may not be able to correctly interpret retrieved lessons because they may be long and not well written 2. Textual representations have multiple interpretations and allow unlimited content having the potential to cause misuse of lessons 3. Textual representations prevent effective verification and validation, they may add even more text Structured lessons (cases): 1. If more than one lesson happens to be retrieved, the representation allows the user to assess its relevance immediately 2. The representation limit the content to what is applicable avoiding multiple interpretations, facilitating correct use 3. The representation can potentially allow automatic verifications, without adding unnecessary content

20. Problems & Solutions (iii) Current Collection: 1. Users may not have the time or skills to submit extended textual lessons 2. Users do not know what specific content to communicate when submitting lessons 3. Users have to compose textual descriptions of their experiences 4. Lack of guidance and misleading instructions result in poorly written lessons Lesson Elicitation Tool: 1. Self-explanatory elicitation with lists to select from make the submission quick and easy 2. LET educates users instructing what contents to communicate 3. A pre-defined format indicates how to communicate experiences by filling out text fields and selecting from drop-down lists 4. Carefully designed guidelines with examples and confirmations results in clear lessons

21. How technology can address Human & Managerial issues Lack of training for knowledge submission Validation of lessons Lack of skills to access the repository Difficulty in interpreting knowledge Enable lesson collection Monitor transfer and oversee reuse Create infrastructure Auto-explanatory tool that orients the submission process Structured format with limited content Active dissemination of knowledge when & where it is applicable Lessons stored in structured that is easily understood Conversational elicitation tool Implement monitored distribution with executable reuse Seek technological solutions