MODULE 3 Documenting Software Surveillance Elements Lesson 1: Documenting Software Surveillance Elements Lesson 2: Risk Assessment Lesson 3: Contract Management Office (CMO) Unique Elements Lesson 4: Customer Unique Elements Lesson 5: Documenting Program Measures

Purpose Given Government contract requirements, Student Guide, and support materials, determine the software surveillance elements documentation requirements using the Software Acquisition Management Instruction (SAMI).

CMQ101 - Course Introduction 6/30/2018 Module Lessons Lesson 1: Documenting Software Surveillance Elements Lesson 2: Risk Assessment Lesson 3: Contract Management Office (CMO) Unique Elements Lesson 4: Customer Unique Elements Lesson 5: Documenting Program Measures

What Will You Learn in This Module? Describes in detail how to document the surveillance elements you are required to engage in when executing the Software Acquisition Management (SAM) mission Describes methods and processes vital to your day-to-day job performance as a Software Professional (SP) 4

SAMI Planning Step Sub – Processes Review The Software Surveillance Process Step 1: Planning Chapter 3, Still Here Chapter 4 Chapter 5 Chapter 3, paragraph 3.18 5

 SAMI Planning: Step 1 Review Step 1 is about Software Surveillance Planning. SAMI planning is broken into the following subtasks: Fact Gathering Documenting Surveillance Tasks/Activities Resource Estimation Finalizing the Software Surveillance Plan (SSP) Issuing Letter(s) of Delegation (LOD)  6

Applicable SAMI Paragraphs Review Documenting Surveillance Tasks/Activities is described in the following SAMI paragraphs: 3.8 Identify and Plan 3.9 Program Measures 3.10 Identify Customer Unique Elements 7

The Identify and Plan Process Review The purpose of the Identify and Plan process is to identify and document: The mandatory level of Software Surveillance Core Tasks/Activities you are required to engage in (predetermined). Any CMO Unique, Customer Unique, and Program Measure Tasks/Activities to be engaged in. 8

The Identify and Plan Process Review, Cont. Mandatory Predetermined Tasks/Activities are: Grouped by Element Categories (also known as Process Areas) SQA, SCM, SM, SSM, PLP/P, SR, SD, C&UT, I&T, FQT, and SWD Further subdivided into performance areas Technical, Cost, and Schedule (Not all Element Categories have Cost/Schedule) Level of engagement is based on the Software Contract Criticality (SCC) Ratings (for DoD work only) SSP Contract Template 9

The SSP Worksheet Review The SSP Worksheet tab within the SSP template is divided into the following planning sections: Core Elements Technical Cost Schedule CMO Unique Elements Customer Unique Elements Program Measures Elements Core Measures Contractually Imposed: Standalone Contractually Imposed: Technical Performance Measures (TPMs) CMO Unique Customer Unique LODs Planned/Issued 10

Lesson 1: Documenting Software Surveillance Elements Objective: Identify documentation required for software surveillance core elements.

CMQ101 - Course Introduction 6/30/2018 Lesson Objective Lesson 1: Documenting Software Surveillance Elements Objective: Identify documentation required for software surveillance core elements.

   Software Contract Criticality (SCC) Rating The SCC is the overall contract rating determined by the Software Professional for: Technical Performance Cost Schedule The SCC Rating influences the planning process and determines the level of DCMA involvement.    13

Core Elements 11 Core Elements: Phased Independent Elements SQA – Software Quality Assurance SCM – Software Configuration Management SM – Software Management SSM – Software Sub-Contract Management PLP/P – Program-Level Processes and Products Phased Dependent Elements SR – Software Requirements SD – Software Design C&UT – Coding and Unit Testing I&T – Integration and Testing FQT – Formal Qualification Testing SWD – Software Delivery 14

Core Elements: Documentation The documentation for each core element will contain the following: Risk Assessment Likelihood Consequence Weighted Risk Rating (WRR) and Rationale Performance Areas Technical Element Cost Element Schedule Element Tasks/Activities To do today: Fill out the SSP!!! SSP Contract Template 15

Core Elements: Documentation, Cont. The following items are the three columns for planning in the template: Optional Surveillance Notes Free text form (manually entered) Task/Activity Surveillance Status Pick list Surveillance Frequency Needed Based on Risk SSP Contract Template 16

Tasks/Activities: Required vs. Not Required For tasks/activities: A gray background means not required. For DoD work, the SCC Rating determines the required tasks/activities (within the Core Technical, Cost and Schedule areas) you must engage in. For Non-DoD work, only those tasks/activities associated with the Acquisition Customer’s requirements (per MOA/MOU, Agreement, etc.) are to be engaged in. 17

Questions 18

Knowledge Review Question 1 Which statement is true about mandatory predetermined tasks/activities? They are grouped by Risk Level They are further subdivided into performance areas Based on the CMO Unique Elements Based on the SCC Rating – For Non-DoD work only

Which is considered to be a performance area? Knowledge Review Question 2 Which is considered to be a performance area? SCC Rating CMO Unique Cost Customer Unique

Lesson Summary Identified the documentation required for software surveillance core elements.

Lesson 2: Risk Assessment Objective: Identify software risk assessments for software surveillance core elements.

CMQ101 - Course Introduction 6/30/2018 Lesson Objective Lesson 2: Risk Assessment Objective: Identify software risk assessments for software surveillance core elements.

Risk Assessment Core elements are risk assessed. SAMI uses the assessment method as described in the Risk Management Guide for DoD Acquisition. Process uses: Likelihood Table Consequence Table Key Process/System Risk Rating (5 x 5) Matrix Risk Management Guide 24

Risk Assessment: Components Likelihood Table Pick list Consequence Table For Cost and Schedule, uses criteria from the SRR tab Weighted Risk Rating (WRR) Numerical value (1–25) representing the degree of risk with 1–11 being low, 12–19 being moderate, and 20–25 being high Automatically calculated based on Likelihood and Consequence index Rationale Based on objective facts (qualitative and/or quantitative) NOT an “If” statement SSP Contract Template 25

Sample Risk Rating and Rationale 26

Sample Risk Rating and Rationale, Cont. 27

Surveillance Frequency Needed Surveillance frequency needed is based on/driven by risk High = Maximum Objective Frequency Moderate = Neither Minimum/Maximum Objective Frequency Low = Minimum Objective Frequency High risk Tasks/Activities will need to be performed more frequently Low risk Tasks/Activities will need to be performed less frequently 28

Questions 29

What is included in the documentation of Risk Assessment? Knowledge Review Question 1 What is included in the documentation of Risk Assessment? Optional Surveillance Notes Weighted Risk Rating Predictability Applicability

Select the true statement about Weighted Risk Ratings. Knowledge Review Question 2 Select the true statement about Weighted Risk Ratings. Numerical value representing the degree of risk, with 1 being the highest and 25 being the lowest Based on subjective information (“if” statement) Automatically calculated based on Likelihood and Consequence index

Lesson Summary Identified the software risk assessments for software surveillance core elements.

Lesson 3: Contract Management Office (CMO) Unique Elements Objective: Describe the process for documenting Contract Management Office unique software surveillance elements.

CMQ101 - Course Introduction 6/30/2018 Lesson Objective Lesson 3: Contract Management Office (CMO) Unique Elements Objective: Describe the process for documenting Contract Management Office unique software surveillance elements.

CMO Unique Elements CMO Unique Element: Any business process you need to execute in order to perform your mission that is not already identified in the Core Elements worksheets for Technical, Cost, and Schedule. Not risk assessed Tasks/Activities are identified Surveillance Engagement is determined by the SP’s judgment and schedule To do today: Document the CMO Unique Elements SSP Contract Template 35

User Identified Tasks/Activities CMO Unique Elements: User Identified Tasks/Activities To add a user identified Task/Activity: Select a code that best matches the Task/Activity Provide a title and description for the Task/Activity Reminder: Document the less common DCMA business practices SSP Contract Template 36

Surveillance Frequency Needed CMO Unique Elements: Surveillance Frequency Needed Based on/driven by Schedule/Requirement/Best Judgment Uses a 12-month schedule “How many times over the course of a 12-month period am I supposed to complete this task?” 37

Questions 38

Which of the following statements is true of CMO Unique Elements? Knowledge Review Question 1 Which of the following statements is true of CMO Unique Elements? Tasks/Activities are always predetermined Risk assessed Identified in the Core Elements worksheet Surveillance Engagement is determined

Knowledge Review Question 2 For CMO Unique Elements, what is the “Surveillance Frequency Needed” based on/driven by? User-Identified Tasks/Activities SP’s Judgment Applicability Column Cost

Lesson Summary Described the process for documenting Contract Management Office unique software surveillance elements.

Lesson 4: Customer Unique Elements Objective: Describe the process for documenting customer unique surveillance elements.

CMQ101 - Course Introduction 6/30/2018 Lesson Objective Lesson 4: Customer Unique Elements Objective: Describe the process for documenting customer unique surveillance elements.

Customer Unique Elements They are not risk assessed Tasks/Activities are identified Surveillance Engagement is determined 44

Identifying Customer Unique Elements To identify customer unique elements: Initiate communications with your customer Discuss/present the draft SSP to your customer, and: Determine whether they have any concerns that may impact YOUR software surveillance planning Determine whether they require any Unique Tasks/Activities or any Unique Measures to be performed Request appropriate acquisition documents (DCMA-INST 203 Section 3.10.5.1) Document pertinent planning information for any Unique Tasks/Activities identified in the “Customer Unique Elements” worksheet 45

Valid Customer Unique Tasks/Activities A Customer Unique Task/Activity is: One that is negotiated with the customer that is NOT already identified in the Core, CMO Unique, or Program Measures (Core/Contractually Imposed) Elements. A VALID Customer Unique Element (Task/Activity) is: Within the scope of the SAM mission Within the competency of the CMO Appropriate based on the SCC 46

Adding Customer Unique Tasks/Activities To add a Customer Unique Task/Activity: Associate the Task/Activity to a category code in the far left column. Pick list Write the Task/Activity description. Select a Task/Activity surveillance status. Surveillance Frequency Needed is based on/driven by the Customer’s Schedule/Specification. Example: Review of a supplier’s software training program. SSP Contract Template 47

Questions 48

What is the last step of identifying Customer Unique Elements? Knowledge Review Question 1 What is the last step of identifying Customer Unique Elements? Initiate communication with the customer. Document pertinent planning information for any Unique Tasks identified in the Customer Unique Elements worksheet. Request the acquisition documents (DCMA-INST 203, Section 3.10.5.1). Discuss/present the draft SSP to your customer.

What makes a Customer Unique Element valid? Knowledge Review Question 2 What makes a Customer Unique Element valid? Outside the scope of the SAM mission Appropriate based on the WRR Must be a predefined task or activity Within the competency of the CMO

Lesson Summary Described the process for documenting customer unique surveillance elements.

Lesson 5: Documenting Program Measures Objective: Describe the process for documenting Program Measures elements.

CMQ101 - Course Introduction 6/30/2018 Lesson Objective Lesson 5: Documenting Program Measures Objective: Describe the process for documenting Program Measures elements.

Program Measures Elements They are not risk assessed Tasks/Activities are identified Surveillance Engagement is determined 54

Planning for Program Measures Identify Program Measures: Core Measures Contractually Imposed – “Standalone” Measures Contractually Imposed – Technical Performance Measures (TPMs) CMO Unique Measures (Optional as determined by the SP) Customer Unique Measures 55

Program Measures: Policies Requirements as listed in DCMA-INST 203: 3.9.1.2 Required Template (Applicable for all measures) 3.9.3.2 SCC High/Moderate (Applicable to DoD only) 3.9.11 Document Data/Measure Specification (Applicable for all measures) 56

Program Measures: Exclusions Exclusions as listed in DCMA-INST 203: 3.9.2 No Measures Required – When the contractual period of performance is 6 months or less, the SP is NOT required to plan for or engage in surveillance activities for Program Measures. 3.9.3.1 SCC Low – When the Technical, Cost, and Schedule SCC Ratings are all LOW, the SP shall use whatever supplier measures are available to attain insight into the health of the program. The SP is NOT required to plan for or engage in surveillance activities for Program Measures. 57

Program Measures: Applicability Column The Applicability Column: Is mapped to different sections of the worksheet, and to other worksheets Is unique to this section Has information in the column that is automatically generated by formulas in the worksheet Has the ability to generate errors that guide the SP to where the error is located Reminder: Errors indicate problems in the SRR tab! SSP Contract Template 58

Program Measures: Contractually Imposed – Standalone Measures These are additional measures required by the contract, that are not already identified as Core Measures. The Contract Reference column is a pick list. The Surveillance Frequency Needed for Contractually Imposed Standalone Measures is based on/driven by Contractual Reporting Requirements/SP’s judgment. Note: The pick list is automatically built by a formula based on the data entered in the Contract Imposed Measures section within the SRR tab; if no data was entered, there will be no data in the pick list. 59

Program Measures: Contractually Imposed – TPMs These are additional measures required by the contract, based on achievement of a selected technical objective. TPMs are sometimes associated with contractual performance goals that, if satisfied by the supplier, often result in an incentive fee/award fee. TPM is normally used in conjunction with Earned Value/ Cost/Schedule indicators (all three must be satisfied). The Contract Reference column is a pick list/dropdown list. The surveillance frequency needed for contractually imposed TPMs is based on/driven by contractual reporting requirements. 60

Program Measures: CMO Unique Measures These are measures not previously identified in the Software Surveillance Core Measures, Contractual Measures, and NOT Customer Unique Measures that the CMO has identified to engage in. While optional, the SP may determine whether additional measures should be self-generated (or obtained, collected, or analyzed) based on the supplier’s historical performance, capability, and/or the SP’s software risk assessment. The surveillance frequency needed for CMO Unique Measures is based on/driven by the SP’s judgment. 61

Program Measures: Customer Unique Measures These are measures that were not previously identified in the Software Surveillance Core Measures, Contractual Measures (i.e. Standalone/TPM), and are NOT CMO Unique. Guiding the acquisition customer to ensure that the need is above and beyond any core, contractually imposed – standalone/TPM, or CMO unique measures, the SP shall determine whether the acquisition customer has any unique measures to be collected, analyzed, and reported. The surveillance frequency needed for customer unique measures is based on/driven by the customer’s schedule/specification. 62

Program Measures: DMAS and DMAR For each measure identified, the SAMI requires implementation of a Data/Measure Analysis Specification (DMAS) and Data/Measure Analysis Results (DMAR) template: One template contains both DMAS and DMAR Worksheets Surveillance Planning: The DMAS tab is used to identify Data/Measure and specify how it will be collected, analyzed, and reported, and specify what actions will be taken when upper/lower thresholds are breached. Surveillance Execution: The DMAR tab is used as the formal document for analysis and reporting. Custom DMAR Worksheet for PAR: This tab is optional; it is NOT required by the SAMI. It is provided solely as a convenience to assist the SP in keeping records of his or her input to a Program Assessment Report (PAR) whenever DCMA instruction 205, Major Program Support (MPS) is applicable. 63

Additional Program Measure Information Later in this course, we will further explore additional Program Measure information such as: Data Collection and Analysis Associated templates Data/Measures Analysis Specification (DMAS) tab Data/Measures Analysis Results (DMAR) tab DMAS/DMAR Template 64

Questions 65

What are Contractually Imposed Standalone Measures? Knowledge Review Question 1 What are Contractually Imposed Standalone Measures? Additional measures required by the contract, based on achievement of a selected technical objective. A measure negotiated with the customer not previously identified. A measure identified by the CMO that was not previously identified. Additional measures required by the contract that are not identified as core measures.

Knowledge Review Question 2 For Contractually Imposed TPMs, what is the Surveillance Frequency Needed based on/driven by? Contractual Reporting Requirements SP’s Judgment Customer’s Schedule Customer’s Specification

Lesson Summary Described the process for documenting Program Measures elements.

Module Summary This module: Identified the documentation required for software surveillance core elements Described the software risk assessment method used to determine surveillance engagement Described the process for documenting Contract Management Office (CMO) Unique software surveillance engagement 69

Module Summary, Cont. This module: Described the process for documenting customer unique surveillance elements. Described the process for documenting Program Measures elements. Workshop #2 70