1. 0 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Effective Use of Lessons Learned: A Key Element for Mission Success Presented by Paul S. Gill Manager, NASA Technical Standards Program Office June 18, 2003 “Try to make original mistakes rather than repeating those of your predecessor.” Secretary of Defense Donald Rumsfeld

2. 1 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge  Lessons Learned Are Crucial To The Agency’s Continued Success  They Capture The Knowledge And Experiences From NASA’s Programs And Projects  A Principal Component Of Any Organizational Culture Committed To Excellence  NASA Recognizes The Need To Systematically Document And Utilize The Knowledge Gained From Past Experiences.  NASA Policy And Guideline (NPG) 7120.5B, “NASA Program And Project Management Processes And Requirements”  NASA Policy Directive (NPD) 8730.3, “NASA’s Quality Management System Policy”  GAO-02-195 Revealed Fundamental Weaknesses In How NASA Performs The Collection And Sharing Of Lessons Learned Information.  Lessons Learned Not Routinely Identified, Collected Or Shared By Program And Project Managers  Lack Of Proper Review For Lessons Learned  Need For More Positive Lessons Learned (Generated Not As a Failure) Background

3. 2 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge  Develop and Initiate a Process to Facilitate the Identification and Input of Lessons Learned Information  Through the utilization of Chartered Technical Discipline Working Groups, Review and Categorize Submitted Lessons Learned Information  Consolidate and Link Lessons Learned Databases That May Exist at Different NASA Facilities and NASA Contractors  Utilize Automated Workflow Capability to Manage and Monitor Tasks  Facilitate the Dissemination of This Information Across a Broad Spectrum Via a Portal Within the Agency’s Knowledge Management Initiative, Thus Providing Programs and Projects With Advanced Search and Retrieval Mechanisms  Obtain Current Performance Metrics to Evaluate System Effectiveness Approach Proposed Initiative Leverages State-of-the-art Information Technologies and Organizational Processes to Optimize the Capture, Management, and Dissemination of Lessons Learned

4. 3 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Lessons Standards and Practices Training Time Update/Create Standards Update/Create Training Communicate Knowledge – Infuse into Practice Projects Generate and Acquire Lessons Generation and Communication of Knowledge Begins With Lessons Project 1Project N Approval Process Gather Lessons at Control Gates

5. 4 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Lessons Learned Implementation Process ATP SRRPDR CDRDCR FRR Post Flight Generate and Communicate Lessons Learned Lessons Learned Databases Technical Discipline Working Group Review PROGRAMS AND PROJECTS (NPG 7120.5) IMPLEMENTATIONFORMULATION Screened And Approved Lessons SYSTEMS ENGINEERING (NPG 71xx.x and SP-6105) CONCEPT DEVELOPMENT DESIGN PHASE DEVELOPMENT & OPERATIONS Proposed Lesson

6. 5 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Programs and Projects Link To Requirements Infuse Into Engineering Process Link the Lessons Learned to System Development Lifecycle Requirements Infuse the Lessons Learned Into the Engineering Process Establish Feedback System Feedbacks Needed to Provide User Based Enhancements COMMUNICATE KNOWLEDGE Generate & Identify Document Categorize Characterize Evaluate Generate & Identify Lessons Learned Document Lessons Learned Categorize Into Discipline Categories Characterize into Programmatic Versus Technical Lessons Learned Validity is Determined by the Working Group GENERATE KNOWLEDGE Lessons Learned Closed-Loop Process

7. 6 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Key Agency Roles Success of a Lessons Learned Program Will Require the Support and Participation of the Following:  Office of the Chief Engineer (Code AE)  Overall Management and Oversight Responsibility (i.e. NPD, NPG, and Program Plan)  Charter the Technical Discipline Working Groups  Chief Information Officer (CIO)  Responsible for Agency Information Technology Infrastructure  Safety and Mission Assurance Office (Code Q)  Responsible for the Collection and Archiving of Mishap Reports and Failure Investigation Reports  Enterprises/NASA Centers/Agency Contractors  Source and Users of the Lessons Learned Information  Membership in Technical Discipline Working Groups  Lesson Learned Program Office  Agency Lessons Learned Working Group  Lessons Learned Information System  Engineering Discipline Working Groups  Responsible for the Collection, Analysis, and Dissemination of Lessons Learned Information  Training (Code FT)  Advanced Program and Project Leadership (APPL)  NASA Engineering Training (NET)

8. 7 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge  Programs and Projects  Generate and Use Knowledge  NASA Technical Discipline Working Groups  Acquire Lessons (Develop, Review, Screen)  Place Lesson in Discipline Taxonomy  Update/create Standards to Incorporate Lessons  NASA Lessons Learned Program  Facilitate Capture and Use of Lessons Learned  Provide On-line Access to Lessons Learned System  NASA Technical Standards Program  Identify Lessons for Potential Conversion Into Technical Standard  Link Lesson to Relevant NASA Preferred Technical Standards  NASA Training  Incorporate Lessons and Standards Into Training Materials  Provide Training on Use of Lessons Learned System Key Program Roles

9. 8 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Lesson Learned Program Proposed Organizational Structure Office of the NASA Chief Engineer Theron Bradley Lessons Learned Executive Gregory Robinson Office of the NASA Chief Engineer Theron Bradley Lessons Learned Executive Gregory Robinson Lessons Learned Program Office NASA Lessons Learned Working Group (Senior Representative From Each Center/JPL) NASA Lessons Learned Working Group (Senior Representative From Each Center/JPL) NASA Engineering Discipline Working Groups Lessons Learned Systems NASA Engineering Management Board

10. 9 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Plans Initiated in FY03:  Enhance Existing Taxonomy of Categories/Subcategories (Technical and Programmatic)  Categorize Existing Lessons Learned Databases or Raster Images (PDF, etc.)  Identify NASA Failure Mishap Reports  Initiate the Development of a Process for Formulating Discipline Working Groups for Screening, Validating and Categorizing Existing Lessons Learned and to Facilitate Searching, and Identifying New Lessons Learned  Initiate and Develop a Common Lessons Learned Process and Format  Lessons Learned To be Infused Into the Agency's Programs and Projects at the Appropriate Stages During the Lifecycle  Inputs From Program and Project Managers Will Help Define the Requirements to Implement Lessons Learned Into Product Lifecycle  Define Agencywide System Architecture Proposed Tasks

11. 10 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Plans for FY04 Through FY08:  Establish an Agency Lessons Learned Working Group  Comprised of Senior Representatives From All NASA Field Facilities and Enterprises  Establish Agencywide Technical Discipline Working Groups  Catalyst for Infusion of Lessons Learned into Product Life Cycle  Working Groups Defined By The Ten Discipline Categories Established by the NASA Technical Standards Program  Develop, Review, and Screen Lessons Learned for Endorsement by the Agency  Working Groups Provide Agencywide Expertise in Specific Technical Areas DISCIPLINE WORKING GROUPS ARE THE KEY TO SUCCESS Proposed Tasks (Cont’d)

12. 11 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Plans for FY04 Through FY08 (Cont’d)  Implementation and Maintenance of the Information System Architecture  Including Advanced Search Engine That Meets System Requirements  Promote the Agency Lessons Learned Initiative Through an Agencywide Awareness and Training Initiative  Designed to Reach All NASA Employees and Supporting Contractors  Ensure Lessons Learned Are Available From a User Friendly Website  Develop Metrics to Ascertain the Effectiveness of the System Usage at the Agency Level  Establish a Feedback System for Inputs Needed to Provide User Based Enhancements Proposed Tasks (Cont’d)

13. 12 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge  NASA Lessons Learned Working Group  NASA Engineering Discipline Working Groups  NASA Lessons Learned Program Website for Agency Wide Use With Timely Feedback Response System  Including Integration of Legacy Lessons Learned Systems  Common Lessons Learned Format  Awareness Initiative  Metrics on Usage of Lessons Learned  A Process Whereby Lessons Learned May Be Linked to the Applicable Period/Phase in the Lifecycle of a Program or Project  Periodic Brief to the NASA Chief Engineer’s Office Deliverables The Anticipated Products of the Proposed Lessons Learned Initiative Are Contained in the Following List:

14. 13 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Challenges to Overcome  Establishment of Technical Discipline Working Groups  Chartered by Code AE  Identify, Categorize, and Promote Incorporation and Use of Lessons Learned  Resources Required (Dollars and Manpower)  Standardization  System Requirements of Standardized Electronic Templates (Data Format Standards) to Facilitate the Capture of Metadata and Integration of Legacy Lessons Learned  Development of Large–Scale Advanced Technology Information System  Get It To the People Agencywide  Education and Awareness on Value of Generating and Using Lessons Learned  Develop Personalization Profile  Infuse Into Program/Project Execution  Oversight of Use in Programs/Projects

15. 14 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge  Lessons Learned /Best Practices Identified  Over 1,700 Lessons Learned/Best Practices Accessible on the NASA Technical Standards Program Website  Over 245 Preferred Technical Standards Are Linked to 477 Unique Lessons Learned  Lessons Learned Sources Have Been Identified From Databases, Technical Papers and Memorandums, Mishap Reports, NASA Special Publications (Monographs)  Lessons Learned Search Screen Capability on NASA Technical Standards Program Website  Over 1 Million Keywords Contained in the Database  Enhanced Search Permits Users to Search by  Multiple Keywords  Phrases  Functional Categories  Organizations  Project  Date Accomplishments To Date

16. 15 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Correlation of Lessons Learned With NASA Technical Standards Taxonomy Discipline Category Title (Several Lessons Learned Fall Into Multiple Categories) Number of Lessons 0000 Documentation and Configuration Management, Program Management 412 1000 Systems Engineering and Integration, Aerospace Environments, Celestial Mechanics 456 2000Computer Systems, Software, Information Systems236 3000Human Factors and Health205 4000 Electrical Systems, Electronics, Avionics/Control Systems, Optics 323 5000 Structures/Mechanical Systems, Fluid Dynamics, Thermal, Propulsion, Aerodynamics 730 6000Materials and Processes, Parts564 7000System Test, Analysis, Modeling, Evaluation673 8000Safety, Quality, Reliability, Maintainability829 9000 Operations, Command, Control, Telemetry/Data Systems, Communications 482 Accomplishments To Date (Cont’d)

17. 16 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Summary  Lessons Learned Are Necessary to Preserve and Enhance the Agency’s Corporate Knowledge  The Effort Must Be Optimized to Meet Users Needs and Fully Take Advantage of Available Information Technology Assets  Success of This Effort Will Be Highly Dependent on Buy-in From All Enterprises/centers  Agency Level Initiative, Including NASA Contractors  NASA Technical Standards Program Office Is Best Positioned to Launch and Lead Effort  Has Effectively Implemented Agency-wide Working Groups  Very Successful Linking Lessons Learned to Technical Standards  Has Understanding of Task/Challenges – Similar to Technical Standards Program, But of Greater Magnitude  A Must Do Task

18. 17 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge This Is Our Opportunity To Make Lessons Learned A Viable Element of the NASA Culture!

19. 18 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Backup

20. 19 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge NPG 71xx.x Systems Engineering Processes and Requirements ConceptDevelopmentProd’nOper’nRetire NPG 7120.5B Program/Project Mgmt. Processes and Requirements Standards Lessons Learned  Capture and Deliver Lessons Learned in their Context of Use  Advanced Search Based on Discipline, Activity, Phase, Etc. Interactive, Web-Based Documents

21. 20 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge StartObservation of Lesson Learned/Best Practice (LL/BP) Submit LL/BP to Lessons Learned (LL) Program Office LL Program Office Initial Screening Submit to Technical Discipline Working Group (TDWG) for Review of Value to NASA Valid? Lesson May Be Sent Back to Author for Editing or Removed From System No Disseminate the Lessons through the Application Server Insufficient Information Not Technically Accurate Duplication Proprietary Information ITAR/EAR Information Lesson Learned Work Flow Lesson May Be Sent Back to Author for Editing or Removed From System Yes NASA Lessons Learned Databases Users Search for Lessons Learned Metrics, Feedback, and Analysis Approved and Recommend Development/Change of a NASA Standard based on LL/.BP? Review Existing Standards for Applicability Yes No

22. 21 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge User Requirements (ICD) & Concept of Operations System Requirements & Architecture Refine Design (SRR, PDR, CDR) Build and Deliver Increments Component Integration and Test System Integration and Test System Demonstration, Verification,and Validation “Vee” Model Of System Development Concept Identification and Integration of Technical Standards and Lessons Learned into Appropriate Stages of the Product Life Cycle

23. 22 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge NASA Standards and Practices Legacy Lessons Learned Legacy Lessons Learned Agency- wide Lessons Learned Knowledge Metadata -Document Content and Structure -Taxonomy and Ontology -Technical Discipline and Discipline Ontology -Etc. External Standards Advanced Common Interface Integrated Access to NASA Knowledge Knowledge-based Search Through Lessons and Standards Discover and Link Related Knowledge Prompt Potential Users That Knowledge Is Available Generate New Knowledge as Part of Project Life Cycle XML-Based Data Exchange In A Common Format A Vision Enabled by Integrated Shared Knowledge  Interactive, Web-based Documents  Deliver Relevant Standards and Lessons Learned in the Context of Use  Gather New Lessons at Critical Project Gates and Engineering Milestones