2. Surveillance Planning Team Presentation to the Engineering Management Council February 21, 20001 Greg Robinson Revision 4

3. Purpose of team To improve probability of mission success for NASA projects by improving Agency guidance for risk-based, functionally integrated surveillance. Surveillance suitable to the project Surveillance proportional to the risk Surveillance that meets project management needs Team Products:  Policy Direction  Procedural Guidance  Streamlined Simplified Tools to Do the Job Team Products:  Policy Direction  Procedural Guidance  Streamlined Simplified Tools to Do the Job

4. Surveillance as a method to improve likelihood of mission success Key objectives:  Integrate surveillance approach:  Engineering, S&MA, Finance, Procurement  Improve effective utilization of surveillance resources  Enhance communication of surveillance results  Surveillance to proactively mitigate risk (part of RBAM)  Make surveillance more response to project management

5. Background (PBC Report) July 1999, Administrator tasked the AA for Procurement to assess how the Agency could improve implementation of Performance Based Contracting (PBC) June 2000, the PBC Team released their report. A Significant Finding:  Many NASA Centers have not developed or implemented contract surveillance plans.  This may be due to the lack of Agency guidance for the development of meaningful surveillance planning. A Significant Finding:  Many NASA Centers have not developed or implemented contract surveillance plans.  This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.

6. Background (NIAT) October 20, NASA Integrated Action Team (NIAT) Report, Enhancing Mission Success - A Framework for the Future NIAT-13 Action Summary:  Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns. NIAT-13 Action Summary:  Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.

7.  Surveillance Planning Team:  Chartered by AA for Procurement and AA for OSMA, November 2000  Also sponsored by the Office of the Chief Engineer  Representation: NASA Centers: JSC KSC MSFC SSC GSFC NASA Headquarters: Code Q Code H Code M Code AE Code S Code Y Team Sponsorship

8.  Current Members: JSC - Jim Nise (SOMO Engineering Management) JSC - Lee Norbraten (Management Systems Office) KSC - Larry Tucci (S&MA) KSC - David Culp (Procurement) KSC- Terry Smith (Quality) MSFC - Barry Musick (Engineering) (for Jim Kennedy) MSFC - Charlie Chesser (S&MA) SSC - Mike Smiles (S&MA) GSFC - Mary Clark (EVM) S - Nancy Porter (Program Office - Space Science) Y - Tom Magner (Program Office - Earth Science) M - John Castellano (Program Office - HEDS) Q - Steve Newman (PBMA) Q - Tom Whitmeyer (Quality) H - Jeff Cullen (Procurement) Team Membership

9. Tools To Be Used Root Cause Analysis Customer Coordination Functional Analysis Benchmarking of Best Practices NIAT and PBC findings analyzed to determine underlying contributing factors. EMC and PMC review. Team determines Engineering, Procurement, S&MA objectives for surveillance, and identify methods for improved implementation and integration. Team validates approach against best-of the best in surveillance activity.

10.  Lack of meaningful guidance for conducting contract surveillance Existing NASA Policy, Procedural Guidance, and FAR Supplementation inadequate Need for continuous risk evaluation of contractor performance(not a one-time plan) “Stove pipe” approach to performing surveillance Contributing Factors Lack of common definition Range of contract mechanisms Involvement of multiple functional organizations Differences in approaches across programs and centers Lack of agency wide training Root Causes Currently Identified

11. Team Strategy CHALLENGES: Large/Diverse Project Base Large/Diverse Functional Support Organizations CHALLENGES: Large/Diverse Project Base Large/Diverse Functional Support Organizations  Define surveillance needs  Provide “risk based” options  Utilize Existing Guidance Where Possible:  NPG 7120.5  NPG 8735.2  NASA FAR and Supplement  Other (EVM etc)

12. Schedule and Products Initial Meeting Agency Objectives Outline for NPD Draft NPD One Week Benchmark Study Benchmark Report Draft NPG For Surveillance Publish Summary of Guidance Examples Jan-00 Feb-00 April-00 Telecon 1/30/00 July-00 Final Report for Chief Engineer Recommended Follow On Actions (Training, Etc.) Meeting 1/9/00 EMC Meeting 2/22/00 Meeting Telecon

13. Current Status - Next Steps  Draft NPD  Survey  NPG Initial Approach (see attached)  Benchmarking Plans  Industry  Agencies  Status meeting with PMC

14. AE/Office of the Chief Engineer is responsible for the NPD, project managers are responsible for initiating and maintaining the surveillance plans. Plans are required for contracts, grants, cooperative agreements and MOUs unless otherwise specified by the Center Director The surveillance plan will: Establish and maintain direct linkage with risk of the contract Have a level of surveillance penetration that is proportional to risk areas and to the scope and risk of work to be performed. Be a collaborative integrated effort with procurement, engineering, S&MA, and finance. Be initiated during development of the SOW Provide definition of government surveillance by type, responsibility, and frequency. Specify if or how sub contract work is addressed. Provide for the communication of surveillance plans and results in a defined manner to appropriate levels of management. NASA engineering, procurement, safety and mission assurance, and other technical organizations shall provide surveillance support to program/project management through collaborative involvement in the development and maintenance of surveillance plans and performance of associated levels of surveillance penetration. Surveillance NPD - Key Elements

15. EMC BACKUP CHARTS

16. “Prime” (PBC) Contractor NASA as Integrator In-House Increasing level of integration responsibility and accountability assigned to contractors 80% PBC Goal Changing Environment

17. Project Management/Risk Management Project Management CostSchedule Technical Performance Contract Management Mission Assurance Safety/ Security Risk Management is a Project Management Tool to Balance Requirements/Resources S&MA Procurement Engineering Functional Organizations Provide Expertise/Information to Support this process Finance

18. Elements of a PBC Contract Performance Requirements and Standards Completion Form Contract Type (FFP, FPIF, CPIF, CPAF -- but not LOE) Well Defined Metrics Contractor Quality Assurance Plan Comprehensive Surveillance Plan Fee for Results Not Best Efforts

19. Surveillance Methods and Tools Control Cause and Effect Analysis Closing a Risk Cost-Benefit Analysis List Reduction Mitigation Status Report Multivoting PERT Chart Problem-Solving Planning Risk Information Sheet Spreadsheet Risk Tracking Stoplight Chart Analyze Pla n Track Baseline Planning Planning Decision Flowchart Planning Worksheet - Brainstorming - Cause and Effect Analysis - Cost-Benefit Analysis - Gantt Charts - Goal-Question-Measure - Interrelationship Digraph - List Reduction - Multivoting - PERT Chart - Work Breakdown Structure Risk Information Sheet Bar Graph Mitigation Status Report Risk Information Sheet Spreadsheet Risk Tracking Stoplight Chart Time Correlation Chart Time Graph Surveillance Plan A Surveillance Plan documents how risks will be monitored on a project: the process, activities, milestones, and responsibilities associated with surveillance. It is a subset of the project plan and is written before the project begins. Identify Baseline Identification and Analysis Contractor provided metrics Contractor Monthly Reports Technical Interface Meetings Milestone Reviews Inspections Assessments Test Reports Financial reports EVM Failure reports Project Metrics Identify Analyze Plan Track Control Communicate

20. Risk Drives Areas of Surveillance

21. Surveillance Proportional To Risk Surveillance Approach Evaluated Risk Based on “Risk Management” Tailored to selected Contractor Tailored to selected Contractor Developed prior to award Developed prior to award Not incorporated into the contract Not incorporated into the contract Insight and Oversight, commensurate with risk Insight and Oversight, commensurate with risk Dynamic

22. Surveillance Suitable to the Project Based on “Risk Management” Dynamic Low Dollar Amount Non Flight No Hazardous Operations High Dollar Amount Flight Hazardous Operations

23. Surveillance Dynamics (RBAM) Final Surveillance Plan (High/Medium Risk) based on selected contractor) ACQUISITION STRATEGY PROCESS Engineering Procurement S&MA, Other REQUIREMENTS POST AWARD SOURCE SELECTION SOLICITATION DEVELOPMENT ACQUISITION STRATEGY DEVELOPMENT Contract Performance RISK RANKING HIGH RISK MEDIUM RISK LOW RISK 22 Project Manager Risk Management Surveillance Results Preliminary Surveillance Plan

24. Level of Surveillance Penetration Mapped on Risk Chart

25. Surveillance Continuum Did they do the right things? Did they do the right things and did they do them right? Increasing technical penetration Review of Processes Review of Process and Implementation Level 0Level 1Level 2Level 3Level 4 No Penetration Level of surveillance contingent on defining an acceptable risk: Technical risk levels Amount of trust in performing organization’s abilities How well processes are defined Level at which NASA is performing Task Agreements for the program Human rating of vehicle Program visibility and impact of failure Design complexity, manufacturing complexity, producibility Value of asset

26. Surveillance Evolves as Risk is Better Defined

27. No Penetration Accept performing organization’s tasks at face value (based on assessment that no penetration is required) Contractor develops and implements verification plan Intermediate Penetration Includes low penetration with addition of daily or weekly involvement to identify and resolve issues Review verification plan, its implementation, and selected verification closure data In-depth Penetration Includes intermediate penetration with addition of: Methodical review of details Independent models to check and compare vendor data, as required Review verification plan, implementation, and concur in all verification closure data Total Penetration Perform a complete and independent evaluation of each task Independent review of all verification documentation (including closure data) and witness verification testing Technical Penetration Levels (by discipline areas) Low Penetration Participate in reviews and Technical Interchange Meetings and assess only the data presented Perform periodic audits on pre-defined process(es) Chair board or serve as board member, or RID writer, at a formal review Participate in resolution and closure of issues Review verification plan and its implementation Level 0 Level 1 Level 2 Level 3 Level 4

28. Intermediate Penetration Includes low penetration with addition of daily or weekly involvement to identify and resolve evolving issues Institute specific, regularly planned status and open action/issues reviews In-depth Penetration Includes intermediate penetration with addition of methodical review of details Government Senior Business Managers may participate in reviews Business Systems Penetration Levels Low Penetration Participate in or perform periodic audits on pre-defined process(es) and/or systems Participate in resolution and closure of issues Hold periodic status and open actions/issues reviews Level 0 Level 1 Level 2 Level 3 Level 4 Total Penetration Perform a complete and independent evaluation of each task Independent review of all Contractor final contract documentation Regularly scheduled meetings of Senior Business Managers (Contractor and Government) to review current/anticipated issues and overall status No Penetration Accept performance at face value (assess that no penetration is required)

29. Technical Surveillance Plan

30. Business Surveillance Plan Estimating System Accounting System Property Mgmt. System Major Sub-K Approval Periodic Floor Audits Proposal development and timely negotiation Procurement System Socio-Economic Performance Labor Relations Special Contract Schedules (LOE or T&M) Performance Measurement Reviews or Audits Corporate Financial Health Performance of audits delegated to DCMA NASA review & consent on all Sub-K action over $1M Union contract negotiations scheduled for this summer. NASA Labor Relations Specialist involved. Fortune 500 Company. No formal surveillance required. Excellent performance to date. Reviewed every 6 Months Monthly review with Senior Managers on status of all Proposal Development activity and negotiation status Performance of audits delegated to DCMA Not applicable Surveillance ApproachSystem/Area