1. Lesson Distribution Gap David W. Aha Rosina Weber Héctor Muñoz-Avila Leonard A. Breslow Kalyan Moy Gupta Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory Booth # 214

2. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Outline Introduction Contributions Context: lessons learned systems, process, organizations Lesson distribution gap How to bridge this gap? Monitored Distribution Example Evaluation, Results Next Steps

3. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Contributions Describe lessons learned process Identify gap in lesson distribution Propose Monitored Distribution Test hypothesis in evaluation Monitored Distribution can improve plan quality Plan evaluator

4. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Knowledge management context Three types of KM initiatives knowledge repositories knowledge access and transfer knowledge environment From Davenport & Prusak’s (1998): Working Knowledge Types of knowledge repositories industry oriented (alert systems, best practices) organization oriented (lessons learned systems) for example,..

5. non-government military government int’l Canadian Army Lessons Learned Centre US Air Force Army Coast Guard Joint Forces Marine Corps Navy non-military int’l European Space Agency Italian (Alenia) French (CNES) Japanese (NASDA) United Nations Department of Energy: SELLS NASA (Ames, Goddard) US Construction Industry Inst. Honeywell GM Hewllet Packard Bechtel Jacobs Company Lockheed Martin E. Sys, Inc DynMcDermott Petroleum Co. Xerox IBM BestBuy Siemens

6. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lessons learned systems Lessons learned systems are repositories of a knowledge artifact called lessons learned KNOWLEDGE ARTIFACTS

7. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lessons learned definition… …or organizational lessons, lessons, lessons identified Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned consists of knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

8. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lessons learned process RETAIN REUSE REVISE RETRIEVE

9. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lessons learned representation indexing elements (case problem) reuse elements (case solution) applicable task preconditions lesson suggestion rationale

10. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

11. Lessons learned example applicable task Installing custom stereo speakers. preconditions The car is the Porsche Boxster. From article “Learning from Mistakes” about Best Buy in knowledge management magazine, April 2001. lesson suggestion Make sure you distinguish the wires leading to the speakers from the wires leading to the side airbag. rationale Somebody has cut the wrong wire because they look alike and the airbag went off with explosive force. This means spending several thousand dollars to replace the airbag in addition to be a potential hazard.

12. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lessons learned process

13. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Lesson distribution methods Broadcasting bulletins, doctrine Passive standalone repository Pull Push Active casting list servers, information gathering tools

14. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Problems with lesson distribution methods  Distribution is divorced from targeted organizational processes.  Users may not know or be reminded of the repository, as they need to access a standalone tool to search for lessons.  Users may not be convinced of the potential utility of lessons.  Users may not have the time and skills to retrieve and interpret textual lessons.  Users may not be able to apply lessons successfully.

15. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned Here is the gap Organization’s members

16. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

17. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

18. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

19. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

20. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

21. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organizational processes Repository of lessons learned How to bridge this gap? Organization’s members

22. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organization’s members Organizational processes Repository of lessons learned How to bridge this gap?

23. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organization’s members Organizational processes Repository of lessons learned How to bridge this gap?

24. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organization’s members Organizational processes Repository of lessons learned How to bridge this gap?

25. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organization’s members Organizational processes Repository of lessons learned Monitored distribution

26. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Organization’s members Organizational processes Repository of lessons learned Monitored distribution Lesson repository is in the same context as targeted processes

27. Problems & Solutions Lessons are distributed to users in the context of the organizational processes. Users don’t need to access a standalone tool. Distribution is divorced from targeted organizational processes. Users need to access a standalone tool to search for lessons. Distribute in the same context does not suffice!

28. Problems & Solutions Users may not have the time and skills to retrieve relevant lessons. Users may not be convinced of the potential utility of lessons. Users may not be able to apply lessons successfully. Intrusive methods may cause more problems than solutions. No significant additional time or skills are required. Users can assess the potential utility of lessons easily. Whenever possible, an ‘apply’ button allows the lesson to be automatically executable. Distribution tightly integrated to the targeted processes.

29. when and where  Distribution tightly integrated to the targeted processes so that lessons are distributed when and where they are needed. Represent lessons as cases (knowledge modeling). Lessons are indexed by their applicability.  Additional benefits are: Case representation facilitates interpretation. Users assess potential utility with lesson rationale. Whenever possible, an ‘apply’ button allows the lesson to be automatically executable. Monitored Distribution Characteristics

30. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Noncombatant Evacuation Operations (NEO) Military operations to evacuate noncombatants whose lives are in danger and rescue them to a safe haven

31. NEO site Assembly Point. Intermediate Staging Base Campaign headquarters safe haven

32. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Example in HICAP HICAP is a plan authoring tool suite Users interact with HICAP by refining an HTN (hierarchical task network) through decompositions http://www.aic.nrl.navy.mil/hicap Muñoz-Avila et al., 1999

33. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA NEO sitesafe haven

34. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Selecting the Suggested Case…

35. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Expanding yields…

36. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA WHY : The enemy might be able to infer that SOF are involved, exposing And the user is notified of a lesson RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY : The enemy might be able to infer that SOF are involved, exposing them. RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY : The enemy might be able to infer that SOF are involved, exposing them. RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY : The enemy might be able to infer that SOF are involved, exposing

37. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA After applying the lesson

38. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Evaluation Hypothesis Using lessons will improve plan quality Methodology Simulated HICAP users generated NEO plans with and without lessons Plan evaluator implemented plans Plan total duration Plan duration before medical assistance Casualties: evacuees, friendly forces, enemies

39. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Plan evaluator non-deterministic (100 plans 10 times each) 30 variables: 12 random e.g., weather, airports length of plans 18 steps e.g., transportation mode, supplies, team size of planning space 3,000,000 13 actual lessons

40. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Plan implementation  Plans where evacuees were transported by land modes have an increased chance of being attacked by enemies.  When an attack happens it increases the number of casualties among evacuees and friendly forces (in proportion to # of evacuees).

41. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Results NEO plan total duration* casualties among evacuees no lessons 39h50 11.48 with lessonsreduction 32h48 8.69 18 % 24 % *The resulting values are averages duration until medical assistance* 29h37 24h13 18 % casualties among enemies 3.083.14 -2 % 9.41 6.57 30 % casualties among friendly forces

42. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Next Steps Collection tool Verify methods, reasoning Integration of informal groups’ and user’s individual features Evaluation with human subjects (simulated users in HICAP) and let human subjects decide on applying lessons Extend MD to other decision support systems and other knowledge artifacts Investigate distribution of experiential knowledge with training knowledge

43. David W. Aha Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory, Washington, DC Rosina Weber Department of Computer Science, University of Wyoming Fall 2001 at Drexel University, PA Héctor Muñoz-Avila Department of Computer Science, University of Maryland Fall 2001 at Leehigh University, PA Leonard A. Breslow Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory, Washington, DC Kalyan Moy Gupta IIT Industries, AES Division, Alexandria, Virginia

44. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

45. Aamodt & Plaza 1994 distribute CBR Cycle and Knowledge Processes

46. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Discussion Intrusive method requires good precision Knowledge representation is costly and so are lives! What’s the worth of 35,000 unused lessons? Knowledge representation can be also support validation Good news: collect lessons into case representation.

47. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Plan evaluator: example lesson applicable task: Assign security element. Conditions for applicability: There are hundreds or more evacuees as to justify a security effort. Lesson suggestion: Recommend that EOD* personnel is utilized in security element. Rationale: Success. EOD two DET ten personnel were employed in a force protection role and assisted USS Nassau security teams in identifying and investigating suspect items brought aboard by evacuees. *EXPLOSIVE ORDNANCE DISPOSAL

48. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA Clarification How is monitored distribution (MD) different from Clippie? In MD, case base task is applicability MD distributes experiential knowledge collected from users in similar roles as the potential reuser Clippie is activated by single word Clippie distributes general instructions/information

49. Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA additional lesson Conditions for applicability: There are representatives of different branches assigned to participate. Lesson suggestion: Assign representatives of all forces to plan. Rationale: Lack of representatives prevent good communication causing delays and miscommunication.