Compliance and Audit Block 17 Block 17

Overview Intent Be able to audit EA products for quality and compliance with the DAF, in order to determine how they can be improved Content Again, using AnnL to assess risks (of an EA as documented) Principles Quality measures Report risks CV3 CV4 In this Block we will see how to assess the risks of emerging technologies, including the price/ performance trade-offs. This material supports 035C1 primarily. This Block draws upon Block 6 Information Collection. The sources of information about emerging technologies, such as www.portal.acm.org are most pertinent here. Of course, it also builds upon Block 9, where we saw how AnnL can be used to develop strategic objectives. Block 17

Activity: What is Compliant? Block 17

Principles of EA Remind them that we have seen Principles for EA before, in Block 5. We can use them as a basis for checking the Compliance of the proposed products with the DAF. Several of the Principles are not all that useful for checking. Most are expressed in general terms - they are values. We need to find the measures that can serve as more precise indicators of the likelihood that these values will not be met by a product. Block 17

Qualities to Audit Use the questions of stakeholders from the DAF Course as values also? What is the difference between these and the principles? Check verification and validation = process risk and product risk Which is what? Principles = process risk Stakeholders = product risk There are additional values that can describe the quality of a proposed EA. What is the difference between the two sets of values? Process risk = quality of the DAF products themselves (verification): logical, well-formed, understandable Product risk = meet the needs (validation): correctly capture capabilities, lead to correct OCT/ FPS Block 17

Using AnnL in Auditing Open In sheet for AnnL for Audit Point out values and measures Block 17

Assess risk of EA products Block 17

Activity: Discuss Remedies Briefly discuss what could be done to reduce the identified shortfalls. Bring out rework effort in redrawing or rethinking design Block 17

Activity: Find risk of own Product As a team For 20 minutes Take one of your products Use AnnL to assess the compliance and quality Be prepared to discuss risks and remedies with others Bring out the point of risk analysis is to find remedies. Ask participants to open AnnL for Audit (pre-populated with values and measures from principles and DAF qualities) Ask team to select one of their own products, get them to set it up as an option in AnnL and then assess it against the measures. Ask them to consider three principles and the three other values that they think have the most impact. After 15 minutes Ask participants to print a risk report (R_11Rsks) Ask one team to highlight risks Ask other team to suggeest what can be done to improve the product in the risk areas. Block 17

Activity: Build CV-3 As a team For 7 minutes Fill in a CV-3 for the risks identified for your own product Block 17

Capability Maturity Model Developed by Software Engineering Institute, Carnegie Mellon University Determines how well an organization uses processes to develop systems Often used to assess suppliers Capability Maturity Models developed by Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh, under contract to the US DOD primarily. Initially had CMM for software, systems engineering, and personnel. Combined them into CMMI-I (Integrated). Used for any process really, including acquisition and now EA (developed in Australia not SEI) Use the framework of CMM as a guide to how well Defence is using EA as an organization. CMMI certification to a certain level is common in tenders for software suppliers (subject to some abuse?) Block 17

CMM Framework Level 1 - Initial Level 2 - Managed (Repeatable) Plans, controls Level 3 - Defined Standards, consistency Level 4 - (Quantitatively) Managed Level 5 - Optimized

Activity: Complete a CV-4 As a team For 10 minutes Rate how well Defence meets the CMM criteria Block 17

Summary Risk management continues with the audit of EA products. We have tools - and forms - for determining these compliance and quality risks Next Block Analysis and presentation Bring out the need to have the best possible products before they are presented to senior officers as part of the capability development process. Block 17