MODULE 8 Executing the Software Surveillance Plan

MODULE 8 Executing the Software Surveillance Plan
Lesson 1: Execution Process Overview Lesson 2: The Process Review Sub-Process Lesson 3: The Software Product Examination Sub-Process Lesson 4: The Formal Review and Audit Sub-Process Lesson 5: The Data Collection and Analysis Sub-Process Lesson 6: The Accept Product Sub-Process Lesson 7: Required Surveillance Documentation for Completing the Software Surveillance Record (SSR) Log

CMQ101 - Course Introduction
5/11/2018 Purpose Given the course materials presented, identify the five basic Software Acquisition Management methods used for engaging in software surveillance.

5/11/2018 Module Lessons Lesson 1: Execution Process Overview Lesson 2: The Process Review Sub-Process Lesson 3: The Software Product Examination Sub-Process Lesson 4: The Formal Review and Audit Sub-Process Lesson 5: The Data Collection and Analysis Sub-Process Lesson 6: The Accept Product Sub-Process Lesson 7: Required Surveillance Documentation for Completing the Software Surveillance Record (SSR) Log

What Will You Learn in This Module?
CMQ101 - Course Introduction 5/11/2018 What Will You Learn in This Module? This module: Describes the tools you will use to analyze the supplier’s performance for a given task or activity Prepares you for the follow-on CMS260 SAM Policy Implementation class

Lesson 1: Execution Process Overview
Objective: Identify the common requirements of the five sub-processes of the execution step of the software surveillance process.

5/11/2018 Lesson Objective Lesson 1: Execution Process Overview Objective: Identify the common requirements of the five sub-processes of the execution step of the software surveillance process.

Execution Process Overview
CMQ101 - Course Introduction 5/11/2018 Execution Process Overview The Software Surveillance Process  Chapter 4 Chapter 3 Chapter 5 Chapter 3, Paragraph 3.18

Applicable SAMI Paragraph Sections
CMQ101 - Course Introduction 5/11/2018 Applicable SAMI Paragraph Sections Executing the SSP consists of five tasks/activities, described in SAMI Chapter 4. Red numbers indicate SAMI Paragraphs. 4.2 4.5 4.3 4.6 4.4

Common Sub-Process Requirements
CMQ101 - Course Introduction 5/11/2018 Common Sub-Process Requirements Engage in the documented surveillance tasks/activities: Identified in Software Surveillance Plan IAW quarterly surveillance schedules Take the appropriate action. Document the results of the surveillance activities. Determine the impact to the SSP.

Take Appropriate Action
CMQ101 - Course Introduction 5/11/2018 Take Appropriate Action Issue a Corrective Action Request (CAR) Issue a Continuous Improvement Opportunity (CIO) Issue a Report As required by customer or For any trend that could impact cost, schedule, or technical performance

5/11/2018 Document Results The SP shall engage in the documented surveillance tasks/activities identified in the Contract, Program, Lite, or Facility SSP(s) in accordance with (IAW) their respective Software Surveillance Schedule(s). The surveillance tasks/activities include: Process Reviews Product Examinations Formal Reviews and Audits Data Collection and Analysis Accept Products

Minimum Documentation of Results
CMQ101 - Course Introduction 5/11/2018 Minimum Documentation of Results Documentation is required for the results of each surveillance activity and typically includes: Supplier name (add link to applicable SSP) Name of task/activity performed Date the task/activity performed Software Professional’s (SP’s) assessment of the process, product, formal review/audit, data, acceptance of product Name of the individual who performed the task/activity

Minimum Documentation of Results, Cont.
CMQ101 - Course Introduction 5/11/2018 Minimum Documentation of Results, Cont. Documentation is required for the results of each surveillance activity and typically includes: How long it took If the task/activity satisfied an audit standard. Surveillance techniques/methods performed Statistics as applicable (e.g., number of attributes reviewed, number observations made, number of defects found, types of defects, and action taken (CAR, CIO, Report, etc.)) Any supporting artifacts

5/11/2018 Required Template Record an entry for all planned and unplanned software surveillance Tasks/Activities performed in the Software Surveillance Record (SSR) Log.

5/11/2018 Determine SSP Impact

Questions

Knowledge Review Question 1
Which document is used to record the results of a surveillance Task/Activity? SSR Log DMAS worksheet DMAR worksheet SSP worksheet of the applicable SSP template

Which statement is correct?
CMQ101 - Course Introduction 5/11/2018 Knowledge Review Question 2 Which statement is correct? You can conduct several Tasks/Activities concurrently. You should perform the Tasks/Activities in sequential order.

5/11/2018 Knowledge Review Question 3 What is the typical sequence of common activities performed during each sub-process of the execution step? Document the results, determine SSP impact, and take action. Determine SSP impact, take action, and document the results. Take action, document the results, and determine SSP impact. Determine SSP impact, document the results, and take action.

Lesson Summary Identified the common requirements of the five sub- processes of the execution step of the software surveillance process.

Lesson 2: The Process Review Sub-Process
Objective: Identify the method of examining a supplier’s process for contractual requirements.

5/11/2018 Lesson Objective Lesson 2: The Process Review Sub-Process Objective: Identify the method of examining a supplier’s process for contractual requirements.

The Process Review Sub-Process (SAMI 4.2)
CMQ101 - Course Introduction 5/11/2018 The Process Review Sub-Process (SAMI 4.2) Use to determine if a supplier’s processes are: Compliant with contractual requirements Being followed Adequate Effective Producing the expected results

Authority and Obligation
CMQ101 - Course Introduction 5/11/2018 Authority and Obligation FAR (a)(40) – “Perform engineering surveillance to assess compliance with contractual terms for schedule, cost, and technical performance in the areas of design, development, and production.” FAR (a)(41) – “Evaluate for adequacy and perform surveillance of contractor engineering efforts and management systems that relate to design, development, production, engineering changes, sub-contractors, tests, management of engineering resources, reliability and maintainability, data control systems, configuration management, and independent research and development.”

Process Review Technique(s)/Method(s)
CMQ101 - Course Introduction 5/11/2018 Process Review Technique(s)/Method(s) Process Proofing Technique Process Compliance Technique Assesses the adequacy and effectiveness of the supplier’s process Assesses the supplier’s adherence to the process Use Process Proofing technique when: Supplier is new to DoD software intensive system development Process has not been proofed Adverse trends had been noted in outcomes Significant changes to an existing process have occurred Use Process Compliance technique when: Prior supplier history has shown process to be effective Process has already been proofed Trend data indicate controlled process that is followed and results in conforming products NOTE: It is possible to do both methods concurrently.

Process Review Technique(s)/Method(s), Cont.
CMQ101 - Course Introduction 5/11/2018 Process Review Technique(s)/Method(s), Cont. Process Proofing Technique Process Compliance Technique Obtain and review requirements, supplier procedures, and/or other supporting artifacts Develop a flowchart/sequence of events (may use supplier’s) Identify key characteristics of the process Evaluate the logical flow Review process and inputs/outputs for compliance (adequacy) Determine depth and scope of review Obtain requirements, supplier procedures, process flowchart, and/or other supporting artifacts Determine key characteristics Evaluate compliance Analyze output data over time

5/11/2018 Upon Completion Once the process review is completed: Take action as appropriate. Document the results. Determine SSP impact.

Questions

5/11/2018 Knowledge Review Question 1 What is a characteristic of a supplier’s process that is assessed by the SP when performing a process review? Testable Adequacy Cost-effectiveness Schedule compliance

Under which criterion would you use the compliance technique?
CMQ101 - Course Introduction 5/11/2018 Knowledge Review Question 2 Under which criterion would you use the compliance technique? Significant changes to an existing process have occurred. The supplier’s process has been effective in the past. Trends had been noted that have resulted in adverse outcomes. The Software Contract Criticality is rated high.

Lesson Summary Identified the method of examining a supplier’s process for contractual requirements.

Lesson 3: The Software Product Examination Sub-Process
Objective: Identify the method of examining a supplier’s product(s) for contractual requirements.

5/11/2018 Lesson Objective Lesson 3: The Software Product Examination Sub-Process Objective: Identify the method of examining a supplier’s product(s) for contractual requirements.

Software Product Examination Sub-Process (SAMI 4.3)
CMQ101 - Course Introduction 5/11/2018 Software Product Examination Sub-Process (SAMI 4.3) Use to determine whether the supplier’s deliverable and non-deliverable products meet contractual requirements: Specifications Formats Content Performance requirements, etc.

5/11/2018 Software Products Any service resulting from a process and/or an output that is either delivered or non-delivered. Examples include: Schedules Work packages SRS/IRS SDD/IDD SDP Code Design artifacts

CMQ101 - Course Introduction 5/11/2018 Authority and Obligation FAR (a)(40) – “Perform engineering surveillance to assess compliance with contractual terms for schedule, cost, and technical performance in the areas of design, development, and production.” FAR (a)(41) – “Evaluate for adequacy and perform surveillance of contractor engineering efforts and management systems that relate to design, development, production, engineering changes, sub-contractors, tests, management of engineering resources, reliability and maintainability, data control systems, configuration management, and independent research and development.”

Software Product Examination Sub-Process Steps
CMQ101 - Course Introduction 5/11/2018 Software Product Examination Sub-Process Steps Software product examination sub-process steps include the following: Determine the product to examine and the examination criteria (Prepare) Conduct the product examination (Engage) May be performed while conducting a Process Review (Streamline)

Determine the Product to Examine (Prepare)
CMQ101 - Course Introduction 5/11/2018 Determine the Product to Examine (Prepare) Determine product examination criteria. Obtain requirements, standards, drawings, Data Item Descriptions (DIDs), etc., for product. Determine key characteristics to review. Determine appropriate examination technique. Examination Technique Description Inspection Examination through observation and professional judgment Testing (not typical) Evaluation by observation, judgment, measurement, analysis Witness Observation of supplier performing inspection and/or test Verification Confirmation by objective evidence that requirements have been fulfilled (test results, documentation, etc.)

Conduct the Product Examination (Engage)
CMQ101 - Course Introduction 5/11/2018 Conduct the Product Examination (Engage) Obtain the product(s) to be examined. Ensure that the product is controlled and uniquely identified. Ensure that the product is consistent with applicable requirements. Determine whether the product meets the requirements as identified in the criteria (within scope of examination).

5/11/2018 Upon Completion Take the appropriate action. Document the results of the surveillance activity. Determine the impact to the SSP.

Questions

An observation of a supplier performing an inspection and/or test describes which examination technique? Inspection Witness Testing Verification

Which technique should you use to determine whether a deliverable or non-deliverable meets specifications, performance requirements, and other contractual requirements? Proofing technique Compliance technique Formal review and audit Product examination

5/11/2018 Knowledge Review Question 3 As you conduct a software product examination, what action should you take? Review materials for validity and adequacy Determine the key characteristics to review Ensure that a product is controlled and uniquely identified Obtain the requirements, standards, drawings, DIDs, etc. for the product

Lesson Summary Identified the method of examining a supplier’s product(s) for contractual requirements.

Lesson 4: The Formal Review and Audit Sub-Process
Objective: Identify the method used to perform a formal review/audit.

5/11/2018 Lesson Objective Lesson 4: The Formal Review and Audit Sub-Process Objective: Identify the method used to perform a formal review/audit.

The Formal Review and Audit Sub-Process (SAMI 4.4)
CMQ101 - Course Introduction 5/11/2018 The Formal Review and Audit Sub-Process (SAMI 4.4) FAR (a)(40) – Perform engineering surveillance to assess compliance with contractual terms for schedule, cost, and technical performance in the areas of design, development, and production. FAR (a)(67) – Support the program, product, and project offices regarding program reviews, program status, program performance, and actual or anticipated program problems. Examples of formal reviews and audits include: System/Software Requirements Review (SRR) Preliminary/Critical Design Review (PDR/CDR) Test Readiness Review (TRR) Physical/Functional Configuration Audit (PCA/FCA)

Formal Reviews and Audits Sub-Process Steps
CMQ101 - Course Introduction 5/11/2018 Formal Reviews and Audits Sub-Process Steps Obtain and review audit requirements (Prepare) Entrance criteria Procedures for conducting the review/audit Related schedules/plans Exit criteria Ensure that supplier is tracking entrance criteria (Monitor) Ensure that progress is made as scheduled Continue to monitor until entry criteria are met EITVOX Model

Formal Reviews and Audits Sub-Process Steps, Cont.
CMQ101 - Course Introduction 5/11/2018 Formal Reviews and Audits Sub-Process Steps, Cont. Participate in formal review/audit (Engage) Process reviews/Product examinations Gather information Review materials for validity/adequacy Raise questions/concerns as appropriate Verify that Action Items (AIs)/minutes are accurately captured Ensure that Supplier is tracking exit criteria and resolving action items (Follow-up) Ensure that supplier is closing out AIs by established due dates Ensure that supplier is resolving issues/defects

Questions

When preparing for a formal review and audit, what should you obtain and review? Bill of lading Test Readiness Review Entrance and exit criteria Program management files

5/11/2018 Knowledge Review Question 2 While monitoring entrance criteria, you find that the supplier is not meeting the schedule and does not have an effective plan of action. How do you respond? Determine the SSP impact. Proceed to participate in the formal review/audit. Reject the product. Take action as appropriate.

Lesson Summary Identified the method used to perform a formal review/audit.

Lesson 5: The Data Collection and Analysis Sub-Process
Objective: Identify the method of performing data collection and analysis.

5/11/2018 Lesson Objective Lesson 5: The Data Collection and Analysis Sub-Process Objective: Identify the method of performing data collection and analysis.

The Data Collection and Analysis Sub-Process (SAMI 4.5)
CMQ101 - Course Introduction 5/11/2018 The Data Collection and Analysis Sub-Process (SAMI 4.5) Use to collect, analyze, and report on data/measures Requirements stability Software size Staffing Defects Earned value CARs, CIOs, etc.

CMQ101 - Course Introduction 5/11/2018 Authority and Obligation FAR (a)(40) – “Perform engineering surveillance to assess compliance with contractual terms for schedule, cost, and technical performance in the areas of design, development, and production.” FAR (a)(41) – “Evaluate for adequacy and perform surveillance of contractor engineering efforts and management systems that relate to design, development, production, engineering changes, sub-contractors, tests, management of engineering resources, reliability and maintainability, data control systems, configuration management, and independent research and development.” PL , Section 804 – “Improvement of Software Acquisition Processes: Requires Defense agencies to develop appropriate metrics for "Performance Measurement“ and include them in contracts.”

Prepare Data Collection and Analysis
CMQ101 - Course Introduction 5/11/2018 Prepare Data Collection and Analysis Obtain the Data/Measures (Prepare) Based on the frequency identified in surveillance schedules May be obtained through: Supplier asset library Program management files IPT meetings Formal/informal reviews Software Quality Assurance (SQA) data Earned Value (EV) Points of Contact (POCs) Software development /management tools Control Account Managers EXPLAIN that this slide describes the DCMA software standard method for conducting data collection and analysis. The SP will use this

5/11/2018 Execution Template – Data/Measure Analysis Specification (DMAS) Worksheet/Tab Used during surveillance planning to: Identify data/measure Specify how data/measure will be: Collected Analyzed Reported Specify what actions will be taken when upper/lower thresholds are breached

Engage Data Collection and Analysis
CMQ101 - Course Introduction 5/11/2018 Engage Data Collection and Analysis Analyze Data/Measures (Engage) Perform analysis in accordance with DMAS tab. Assess data/measures over time for trends. Perform comparative analysis on multiple related data/measures. DMAS/DMAR

5/11/2018 Execution Template – Data/Measure Analysis Results (DMAR) Worksheet/Tab Used during surveillance execution Formal document for analysis and results DMAS/DMAR

Questions

Select the worksheet and section which contains the data to be collected and analyzed during software surveillance. SSP Worksheet; Program Measures SSP Worksheet; Software Requirements SRR Worksheet; Contractually Imposed Measures SRR Worksheet; Software Deliverables/Non-Deliverables

Complete the following statement. A DMAS does not _________.
Knowledge Review Question 2 Complete the following statement. A DMAS does not _________. Specify how data/measures will be collected, analyzed, and reported Define the data/measures to be collected, analyzed, and reported Specify actions taken when thresholds are breached Identify data/Analyze results

5/11/2018 Knowledge Review Question 3 You must report the results for each measure that is collected and analyzed on a separate DMAR worksheet. True False

Lesson Summary Identified the method of performing data collection and analysis.

Lesson 6: The Accept Product Sub-Process
Objective: Identify the method of performing product(s) acceptance.

5/11/2018 Lesson Objective Lesson 6: The Accept Product Sub-Process Objective: Identify the method of performing product(s) acceptance.

The Accept Product Sub-Process (SAMI 4.6)
CMQ101 - Course Introduction 5/11/2018 The Accept Product Sub-Process (SAMI 4.6) Use to accept software-related products as applicable Software code or application Software documentation: Software Development Plan (SDP) Software Configuration Management Plan (SCMP) Software Quality Assurance Plan (SQAP), etc.

Who May Accept Software Product(s)
CMQ101 - Course Introduction 5/11/2018 Who May Accept Software Product(s) FAR Responsibility for Acceptance – Acceptance of supplies or services is the responsibility of the Contracting Officer. When this responsibility is assigned to a cognizant contract administration office or to another agency (see (g)), acceptance by that office or agency is binding on the Government. Quality Assurance normally accepts products as the DCMA authority. See DCMA-INST 314, Product Acceptance and Release Paragraph Software Professionals having SPDP certification are authorized to accept products.

5/11/2018 Accept Product Steps Verify time, method, and place of acceptance (Supplier Request) Verify time Determine method (e.g., DD 250, Bill of Lading) Identify place of acceptance Determine acceptance criteria (Prepare) As specified in contract

Accept Product Steps, Cont.
CMQ101 - Course Introduction 5/11/2018 Accept Product Steps, Cont. Evaluate product against acceptance criteria (Engage) Ensure that product is under Configuration Management. Determine acceptability of product. Verify that the documentation trail and supporting data are complete.

Accept Product Steps, Cont.
CMQ101 - Course Introduction 5/11/2018 Accept Product Steps, Cont. Accept product (Accept) IF the product: THEN the SP shall: Meets acceptance criteria Accept the product (DCMA-INST 314) Is being accepted by someone other than the SP (whether standalone, embedded in an end system/item, or firmware) Indicate the acceptability of the product to the person accepting it Does not meet acceptance criteria Reject the product

Questions

Who should contact the SP when a system/subsystem is being accepted with new or updated embedded software? CMO Delegating Software Professional Quality Assurance SPDP-certified Software Professional

During the steps of the accept product sub-process, the Software Professional would verify all of the following information EXCEPT the: Exit criteria Method of acceptance Place of acceptance Time of acceptance

Lesson Summary Identified the method of performing product(s) acceptance.

Lesson 7: Required Surveillance Documentation for Completing the Software Surveillance Record (SSR) Log Objective: Identify the required surveillance documentation for completing the SSR Log.

5/11/2018 Lesson Objective Lesson 7: Required Surveillance Documentation for Completing the Software Surveillance Record (SSR) Log Objective: Identify the required surveillance documentation for completing the SSR Log.

Documentation The Software Surveillance Record (SSR) log is a standalone document used by the SP containing the detailed results of the task or activity performed. SSR Log attributes include: Task and activity performed Link to applicable Software Surveillance Plan (SSP), project name, or program PLAS code Notes Date performed Name of SP(s)

5/11/2018 The SSR Log Record your surveillance activities in the SSR Log. SSR Log

The SSR Log Findings Section
CMQ101 - Course Introduction 5/11/2018 The SSR Log Findings Section The FAR requires the DCMA to maintain records for: Number of observations made Number of defects Types of defects Attributes

5/11/2018 FAR FAR Contract administration office responsibilities When a contract is assigned for administration to the contract administration office cognizant of the contractor’s plant, that office, unless specified otherwise, shall… (c) Maintain, as part of the performance records of the contract, suitable records reflecting— (1) The nature of Government contract quality assurance actions, including, when appropriate, the number of observations made and the number and type of defects;…

5/11/2018 Types of Defects Determine the types of defects based on the results of the activity performed FAR Definitions: Critical Nonconformance: hazardous or unsafe conditions for individuals using, maintaining, or depending upon the supplies or services; or is likely to prevent performance of a vital agency mission Major Nonconformance: failure of the supplies or services, or to materially reduce the usability of the supplies or services for their intended purpose Minor Nonconformance: materially reduce the usability of the supplies or services for their intended purpose, or depart from established standards having little bearing on the effective use or operation of the supplies or services SSR Helper Guide

Questions

What information should be recorded in the SSR Log?
CMQ101 - Course Introduction 5/11/2018 Knowledge Review Question 1 What information should be recorded in the SSR Log? Surveillance method performed Surveillance period covered Prime contract number Date of report

Which reference would you use to determine how to classify a defect?
CMQ101 - Course Introduction 5/11/2018 Knowledge Review Question 2 Which reference would you use to determine how to classify a defect? SAMI SRR Log Helper Guide DMAS/DMAR Helper Guide Federal Acquisition Regulation (FAR)

Lesson Summary Identified the required surveillance documentation for completing the SSR Log.

5/11/2018 Module Summary This module covered: Execution Process Overview The Process Review Sub-Process The Software Product Examination Sub-Process The Formal Review and Audit Sub-Process The Data Collection and Analysis Sub-Process The Accept Product Sub-Process Required Surveillance Documentation for Completing the Software Surveillance Record (SSR) Log Workshop #6 Lesson 4: Safety Stock

MODULE 8 Executing the Software Surveillance Plan

Similar presentations

Presentation on theme: "MODULE 8 Executing the Software Surveillance Plan"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

MODULE 8 Executing the Software Surveillance Plan

Similar presentations

Presentation on theme: "MODULE 8 Executing the Software Surveillance Plan"— Presentation transcript:

Similar presentations

About project

Feedback