1. Planning for and Managing Software Verification & Validation (V&V) Quality Assurance Project Oversight Jeff Lewis, PMP The Æon Group, Inc. Jeff.Lewis@TheAeonGroup.net jlewis@ieee.org

2. 2 Planning for and Managing VV QA POC J. Lewis References Materials used in developing this presentation –Institute of Electrical and Electronics Engineers (IEEE) www.ieee.org IEEE Standard 1012-1998, IEEE Standard for Software Verification and Validation IEEE Standard 1058-1998, IEEE Standard for Software Project Management Plans (this Standard has been updated by ISO/IEC/IEEE 16326-2009) IEEE Standard 12207, Standard for Information Technology – Software Life Cycle Processes (this Standard has been updated by ISO/IEC/IEEE 15288-2008) –State of California www.ocio.ca.gov California Project Management Methodology (CA-PMM) California Project Oversight Framework –Carnegie Mellon University Software Engineering Institute (SEI) www.sei.cmu.edu Software Capability Maturity Model (CMM) –Project Management Institute www.pmi.org A Guide to the Project Management Body of Knowledge

3. 3 Planning for and Managing VV QA POC J. Lewis Agenda Background & Definitions –Software Verification & Validation –Quality Assurance –Oversight –Level (Degree) of Independence –Integrity Levels Comparison of V&V, Oversight, & QA –Focus Planning for & Managing V&V, Oversight, & QA –Defining Tasks (Scope) for V&V, Oversight, & QA –Considerations for Managing V&V, Oversight, & QA Summary

4. 4 Planning for and Managing VV QA POC J. Lewis Background (Foundations)

5. 5 Planning for and Managing VV QA POC J. Lewis Project Management Plan Processes Management Methods & Processes –Startup, Work, Control, Closeout –Change –Risk –Reviews –Reporting –Communications –Human Resources Technical Methods & Processes –Methods for producing product Supporting Plans & Processes –Scope –Requirements –Schedule –Cost –Verification and validation –Configuration management –Quality assurance –Software documentation –Reviews and audits –Problem resolution –Subcontractor management –Process improvement –Procurement

6. 6 Planning for and Managing VV QA POC J. Lewis Software V&V

7. 7 Planning for and Managing VV QA POC J. Lewis Software Verification & Validation Software Verification process provides objective evidence whether the software and its associated products and processes –Conform to requirements for all life cycle activities during each life cycle process –Satisfy standards, practices, and conventions during life cycle processes –Successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities Software Validation process provides evidence whether the software and its associated products and processes –Satisfy system requirements allocated to software at the end of each life cycle activity –Solve the right problem –Satisfy intended use and user needs IEEE Standard 1012-1998

8. 8 Planning for and Managing VV QA POC J. Lewis V&V Tasks by Life Cycle Phase IEEE Standard 1012-1998

9. 9 Planning for and Managing VV QA POC J. Lewis Optional V&V Tasks Algorithm Analysis Audit Performance Audit Support Control Flow Analysis  Cost Analysis Database Analysis Data Flow Analysis Disaster Recovery Plan Assessment  Distributed Architecture Assessment  Feasibility Study Evaluation Independent Risk Assessment Inspection Operational Evaluation Performance Monitoring Post Installation Validation  Project Management Oversight Support Qualification Testing Regression Analysis and Testing Reusability Assessment  Security Assessment Simulation Analysis  Sizing and Timing Analysis System Software Assessment Test Certification Test Evaluation  Test Witnessing Training Document Evaluation User Documentation Evaluation User Training V&V tool plan generation  Walkthroughs –Design –Requirements –Source code –Test IEEE Standard 1012-1998

10. 10 Planning for and Managing VV QA POC J. Lewis Quality Assurance

11. 11 Planning for and Managing VV QA POC J. Lewis Quality Assurance Multiple Definitions & Contexts –Quality Management –Quality Control –Quality Assurance Quality –Consistently meeting or exceeding customer expectations –The degree to which a set of inherent characteristics fulfill requirements

12. 12 Planning for and Managing VV QA POC J. Lewis Software Quality Assurance Provides management with appropriate visibility into the process being used and the products being built Verifies compliance with project requirements Involves –Reviewing or auditing products and processes –Providing appropriate visibility to affected groups and Senior Management –Identifying, documenting, and tracking noncompliance issues Based on a documented plan and processes

13. 13 Planning for and Managing VV QA POC J. Lewis SQA Functions Ensure SQA activities exist for all software development and maintenance projects Objectively and independently verify that all software products and activities adhere to the applicable standards, procedures, and requirements Inform all projects of SQA activities and results Seeks to resolve non-compliance issues with the software project –For issues not resolvable within the software project, the SQA function escalates the issue to an appropriate level of management Acts in an advisory role in clarifying and understanding applicable software procedures and practices

14. 14 Planning for and Managing VV QA POC J. Lewis SQA Methods & Techniques Audit –Process Compliance Project Management Supporting Roles (CM, IV&V, Contracts, Documentation) –Checklist Product Development Peer Reviews Collect Statistics –Compliance –Rework –Process Effectiveness

15. 15 Planning for and Managing VV QA POC J. Lewis QA for a Software Project 1. Develop Plans –Project Management Plan Development Processes Management Processes Supporting Processes –Risk Management –Configuration Management –Quality Assurance Plan 2. Follow Plans –Process Audits –Compliance Checklists 3. Measure Success –Metrics

16. 16 Planning for and Managing VV QA POC J. Lewis Process Compliance Checklist (Example) Project Management Plan Process Compliance –Has the Project Management Plan (PMP) been approved by the ITEC and customer? –Does the PMP follow ANSI/IEEE 1058-1998? –Does the PMP clearly state the project deliverables? –Does the PMP clearly identify project start and finish dates? –Does the PMP contain a project charter and project objective statement? –Is the approved PMP entered in the CM library? –...

17. 17 Planning for and Managing VV QA POC J. Lewis Architecture Design Review Checklist (Example) 1.1 Context Diagram 1.1.1 Has every external device, software system, or subsystem with which your system must communicate been represented as a terminator? List those which have not. 1.1.2 One of the following statements should be true for each terminator on the context diagram. The terminator represents a low-level physical interface (i.e., an input/output device with little or no potential for concurrent access), or The terminator represents a software system satisfying requirements not allocated to your software configuration item (SCI), but with which your SCI is required to interact. No hardware interface is needed to connect your SCI with the other software system.

18. 18 Planning for and Managing VV QA POC J. Lewis Detailed Design Review Checklist (Example) Has the development team demonstrated that every requirement is traceable to the detailed design? Is the detailed design valid (complete, consistent, feasible, maintainable, testable? Are the detailed design and critical parameter budgets (e.g., sizing, timing, accuracy) for the program within the limits given in the System Specification, and additionally, within the limitations of the physical and functional environments? Are current, detailed implementation plans and test plans adequate and complete? Are there any critical issues (e.g., any requirements not satisfied), and what is required to resolve these issues?

19. 19 Planning for and Managing VV QA POC J. Lewis Indicators (Example)

20. 20 Planning for and Managing VV QA POC J. Lewis Metrics (Example) Average = 5.7 weeks Target < 8 weeks

21. 21 Planning for and Managing VV QA POC J. Lewis Project Oversight

22. 22 Planning for and Managing VV QA POC J. Lewis Project Oversight Project oversight is defined as “an independent review and analysis to determine if the project is on track to be completed within the estimated schedule and cost, and will provide the functionality required by the sponsoring business entity. Project oversight identifies and quantifies any issues and risks affecting these project components.” Related Terms –V&V –Graduated Oversight –Review, Assess, Report, Track Project Oversight Framework

23. 23 Planning for and Managing VV QA POC J. Lewis Project Oversight 3 fundamental types –Technical Project Oversight V&V (IEEE Std 1012-1998) QA of Technical Performance QA of Technical Product –Management Project Oversight QA of Project Management Function –Processes & Products of Project Management Project Oversight Framework –Special Project Reporting In response to external stakeholder –e.g. Sponsor, Control Agency

24. 24 Planning for and Managing VV QA POC J. Lewis Level of Independence

25. 25 Planning for and Managing VV QA POC J. Lewis Level (Degree) of Independence No Independence –Part of the development organization Managerial Independence –Separate from the development and program management organizations –Selects the segments of the project/product to subject to oversight Technical Independence –Utilizes personnel who are not involved in the development of the software Financial Independence –Control of the oversight budget is vested outside of the development organization IEEE Standard 1012-1998

26. 26 Planning for and Managing VV QA POC J. Lewis Software Integrity Levels

27. 27 Planning for and Managing VV QA POC J. Lewis Software Integrity Levels Integrity levels are –used as an objective basis for quantifying product risk –mandatory (any scheme may be used) A good example is shown in Annex B of IEEE Std 1012-1998 (A Software Integrity Level Scheme) –updated at each life-cycle phase (not static) as part of criticality analysis task –established at the beginning of the project documented in the project SVVP –used as the basis of V&V planning assigned to each function (requirement, component, function, etc.) –documented in a V&V Task Report and the V&V Final Report adjusted based on developer methods and risk plans IEEE Standard 1012-1998

28. 28 Planning for and Managing VV QA POC J. Lewis Example Software Integrity Level Scheme Based on intended use and application of the software product Integrity levels are used to quantify software product criticality

29. 29 Planning for and Managing VV QA POC J. Lewis Failure Consequences CATASTROPHIC CRITICAL MARGINAL NEGLIGIBLE Loss of human life; complete mission failure; loss of system security; extensive financial or social loss Major or permanent injury; partial loss of mission; major financial or social loss Severe injury or illness; degradation of secondary mission; some financial or social loss Minor injury or illness; minor impact on system performance; operator inconvenience CONSEQUENCE DESCRIPTION

30. 30 Planning for and Managing VV QA POC J. Lewis Software Integrity Levels Key Features Software Integrity Levels… Objective Scalability –Failure Consequences –Intended Use of Software –V&V Rigor to be Applied –Minimum V&V Tasks via Integrity Levels –Defines exit criteria for each task Levels of V&V Independence Optional V&V Tasks

31. 31 Planning for and Managing VV QA POC J. Lewis Example – Integrity Level Assignment Notes: Assigned to all functions Entire solution receives integrity level of highest sub-function NOT a static assignment –Only point-in-time based on present information Recorded in V&V Task Report 3 1 11 2 2 2 22 2 2 2 33 2 3 3 Integrity Levels 4 = High 3 = Major 2 = Moderate 1 = Low 0 = None

32. 32 Planning for and Managing VV QA POC J. Lewis 3 1 11 2 2 2 22 2 2 2 33 2 3 3 Integrity Level Assignment… Through the Project Lifecycle 3 1 11 2 2 2 22 2 2 2 33 2 3 3 3 1 11 2 2 2 22 2 2 2 33 2 3 3 3 1 11 2 2 2 22 2 2 2 33 2 3 3 3 1 11 2 2 2 22 2 2 2 33 2 3 3 Concept e.g. in ConOps or FSR O&M e.g. In Ops Doc’s

33. 33 Planning for and Managing VV QA POC J. Lewis Comparison of V&V QA Oversight

34. 34 Planning for and Managing VV QA POC J. Lewis Related of Terms & Concepts Project Oversight Graduated Oversight –Project Complexity Independence –From procuring/managing department V&V Integrity Levels –Product Failure Consequences Independence –From developer

35. 35 Planning for and Managing VV QA POC J. Lewis IV&V vs QA IV&V –Occurs in parallel to development activities Part of design team is optimal interface point  Independent systems engineering analysis of methods and products Systems engineering context –Verified compliance with IEEE Standard 1012-1998 Quality Assurance –Product evaluation Compliance with standards, outlines, documentation guidelines  Process compliance audit Repeatability, measure of compliance –Usually within development organizational structure –Compliant with IEEE Standard 730 (QA Plans)

36. 36 Planning for and Managing VV QA POC J. Lewis IV&V vs Project Oversight IV&V –Systems engineering focus  Objective to build quality into system solution during development –Involvement is based on assigned integrity level of product –Degree of independence Project Oversight –Focused on risk reporting & special studies  Objective is to mitigate risk through reporting to higher level –Lower level of involvement Monthly, Quarterly –Degree of independence

37. 37 Planning for and Managing VV QA POC J. Lewis IV&V vs Project Management Oversight IV&V  Systems engineering focus –Focus on fitness for use of final product Project Management Oversight  Project management focus Is the project following defined project management processes? –Focus on cost and schedule compliance

38. 38 Planning for and Managing VV QA POC J. Lewis Scoping & Planning V&V QA Oversight

39. 39 Planning for and Managing VV QA POC J. Lewis How to Scope and Cost Technical Oversight Integrity Assessment Perform analysis to each function of the solution Assign minimum V&V tasks according to software integrity level assignments Estimate V&V tasks according to rigor necessary - apply costs via rate structure and/or contract definition Select additional V&V tasks (more minimum tasks + optional tasks) in response to project risk areas Assign / Adopt the software integrity level scheme and... Risk Assessment Document Management Risks etc (SEI Risk Taxonomy)

40. 40 Planning for and Managing VV QA POC J. Lewis Minimum V&V Tasks Assigned by Integrity Level Level 4 – High –All V&V tasks are applied Level 3 – Major Level 2 – Moderate Level 1 – Low –Minimum tasks focus on classical risk areas

41. 41 Planning for and Managing VV QA POC J. Lewis How to Scope and Cost Project Oversight Determine Scope of Project Oversight –Oversight tasks are defined commensurate with the complexity of the project Oversight Framework –QA tasks are defined in the QA Plan, defined in response to the suite of management plans created for the project Determine Cost of Project Oversight –Apply costs via rate structure or contract definition

42. 42 Planning for and Managing VV QA POC J. Lewis How to Scope and Cost Quality Assurance Develop Management & Technical Plans (Predecessor) Define Quality Assurance Methods Commensurate with Management and Technical Plans –Methods Product Conformance Process Compliance –Reporting (Including Degree of Independence) Establish Estimates From Work Breakdown –Number of and Timing for Products –Management and Technical Process Model

43. 43 Planning for and Managing VV QA POC J. Lewis Managing V&V QA Oversight

44. 44 Planning for and Managing VV QA POC J. Lewis Considerations for Managing V&V, QA, and Oversight The oversight entity (entities) should be placed high enough in the organization to be –Free from undue influence –Reporting to a decision maker In order to be effective, the oversight entity (entities) must be tasked in a manner timely to the process to be able to influence the quality of the results of the process The oversight entity (entities) must possess the knowledge, skills, and abilities to perform its tasks effectively and efficiently

45. 45 Planning for and Managing VV QA POC J. Lewis Summary

46. 46 Planning for and Managing VV QA POC J. Lewis Summary V&V, QA, & Oversight –Are similar –Can overlap –Evaluate technical and/or management products/performance –Should provide decision makers with the right information at the right time to enable informed decision making

47. 47 Planning for and Managing VV QA POC J. Lewis Questions ?