1. Compliance and Audit
Block 17
Block 17

2. Overview
Intent
Be able to audit EA products for quality and compliance with the DAF, in order to determine how they can be improved
Content
Again, using AnnL to assess risks (of an EA as documented)
Principles
Quality measures
Report risks
CV3
CV4
In this Block we will see how to assess the risks of emerging technologies, including the price/ performance trade-offs. This material supports 035C1 primarily.
This Block draws upon Block 6 Information Collection. The sources of information about emerging technologies, such as are most pertinent here.
Of course, it also builds upon Block 9, where we saw how AnnL can be used to develop strategic objectives.
Block 17

3. Activity: What is Compliant?
Block 17

4. Principles of EA
Remind them that we have seen Principles for EA before, in Block 5. We can use them as a basis for checking the Compliance of the proposed products with the DAF.
Several of the Principles are not all that useful for checking.
Most are expressed in general terms - they are values.
We need to find the measures that can serve as more precise indicators of the likelihood that these values will not be met by a product.
Block 17

5. Qualities to Audit
Use the questions of stakeholders from the DAF Course as values also?
What is the difference between these and the principles?
Check verification and validation = process risk and product risk
Which is what?
Principles = process risk
Stakeholders = product risk
There are additional values that can describe the quality of a proposed EA. What is the difference between the two sets of values?
Process risk = quality of the DAF products themselves (verification): logical, well-formed, understandable
Product risk = meet the needs (validation): correctly capture capabilities, lead to correct OCT/ FPS
Block 17

6. Using AnnL in Auditing Open In sheet for AnnL for Audit
Point out values and measures
Block 17

7. Assess risk of EA products
Block 17

8. Activity: Discuss Remedies
Briefly discuss what could be done to reduce the identified shortfalls.
Bring out rework effort in redrawing or rethinking design
Block 17

9. Activity: Find risk of own Product
As a team
For 20 minutes
Take one of your products
Use AnnL to assess the compliance and quality
Be prepared to discuss risks and remedies with others
Bring out the point of risk analysis is to find remedies.
Ask participants to open AnnL for Audit (pre-populated with values and measures from principles and DAF qualities)
Ask team to select one of their own products, get them to set it up as an option in AnnL and then assess it against the measures. Ask them to consider three principles and the three other values that they think have the most impact.
After 15 minutes
Ask participants to print a risk report (R_11Rsks)
Ask one team to highlight risks
Ask other team to suggeest what can be done to improve the product in the risk areas.
Block 17

10. Activity: Build CV-3 As a team For 7 minutes
Fill in a CV-3 for the risks identified for your own product
Block 17

11. Capability Maturity Model
Developed by Software Engineering Institute, Carnegie Mellon University
Determines how well an organization uses processes to develop systems
Often used to assess suppliers
Capability Maturity Models developed by Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh, under contract to the US DOD primarily.
Initially had CMM for software, systems engineering, and personnel. Combined them into CMMI-I (Integrated).
Used for any process really, including acquisition and now EA (developed in Australia not SEI)
Use the framework of CMM as a guide to how well Defence is using EA as an organization.
CMMI certification to a certain level is common in tenders for software suppliers (subject to some abuse?)
Block 17

12. CMM Framework Level 1 - Initial Level 2 - Managed (Repeatable)
Plans, controls
Level 3 - Defined
Standards, consistency
Level 4 - (Quantitatively) Managed
Level 5 - Optimized

13. Activity: Complete a CV-4
As a team
For 10 minutes
Rate how well Defence meets the CMM criteria
Block 17

14. Summary Risk management continues with the audit of EA products.
We have tools - and forms - for determining these compliance and quality risks
Next Block
Analysis and presentation
Bring out the need to have the best possible products before they are presented to senior officers as part of the capability development process.
Block 17