SQM - 1DCS - ANULECTURE 11-2005 Software Quality Management Software Quality Management Processes V & V of Critical Software & Systems Ian Hirst.

Slides:



Advertisements
Similar presentations
Integra Consult A/S Safety Assessment. Integra Consult A/S SAFETY ASSESSMENT Objective Objective –Demonstrate that an acceptable level of safety will.
Advertisements

2009 – E. Félix Security DSL Toward model-based security engineering: developing a security analysis DSML Véronique Normand, Edith Félix, Thales Research.
Software Quality Assurance Plan
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Processes.
Software Engineering CSE470: Process 15 Software Engineering Phases Definition: What? Development: How? Maintenance: Managing change Umbrella Activities:
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Processes.
Software Processes Coherent sets of activities for specifying, designing, implementing and testing software systems.
Ask Pete Acquired Software Knowledge Project - Estimation- Tool - Effort Presented to the NASA OSMA SAS ‘01 NASA IV&V Facility September 5-7, 2001 Tim.
1 Independent Verification and Validation Current Status, Challenges, and Research Opportunities Dan McCaugherty IV&V Program Manager Titan Systems Corporation.
Rational Unified Process
R R R CSE870: Advanced Software Engineering (Cheng): Intro to Software Engineering1 Advanced Software Engineering Dr. Cheng Overview of Software Engineering.
Software Engineering General Project Management Software Requirements
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Process Models.
©Ian Sommerville 2000 Software Engineering, 6th edition Slide 1 Software Processes l Coherent sets of activities for specifying, designing, implementing.
1 Introduction to System Engineering G. Nacouzi ME 155B.
Trade Study Training Need and Goals Need Consistent methodologies and practices performing trade studies Pros/cons, advantages/disadvantages, customer/management.
COMP8130 and 4130Adrian Marshall 8130 and 4130 Test Management Adrian Marshall.
Stoimen Stoimenov QA Engineer QA Engineer SitefinityLeads,SitefinityTeam6 Telerik QA Academy Telerik QA Academy.
Introduction to Software Testing
Software Verification and Validation (V&V) By Roger U. Fujii Presented by Donovan Faustino.
NASA IV&V Facility Software Independent Verification and Validation (IV&V) NASA IV&V Facility Fairmont, West Virginia Judith N. Bruner Acting Director.
1 First, some interesting numbers: ~2,000 ~80 2 >200 [To be defined on March 23]
SOFTWARE QUALITY ASSURANCE Asst. Prof. Dr. Selim BAYRAKLI Maltepe University Faculty of Engineering SE 410.
Chapter 3 Software Processes.
Miguel Nunes Information Systems Project Management IS Project Resources.
What is Business Analysis Planning & Monitoring?
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 18 Slide 1 Software Reuse.
S/W Project Management
Project Risk Management. The Importance of Project Risk Management Project risk management is the art and science of identifying, analyzing, and responding.
Chapter 6 Software Implementation Process Group
Test Organization and Management
Chapter 11: Project Risk Management
Dillon: CSE470: SE, Process1 Software Engineering Phases l Definition: What? l Development: How? l Maintenance: Managing change l Umbrella Activities:
VTT-STUK assessment method for safety evaluation of safety-critical computer based systems - application in BE-SECBS project.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Processes.
T. Dawson, TASC 9/11/13 Use of a Technical Reference in NASA IV&V.
©Ian Sommerville 2000, Mejia-Alvarez 2009 Slide 1 Software Processes l Coherent sets of activities for specifying, designing, implementing and testing.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 1 Slide 1 Software Processes (Chapter 3)
Topic (1)Software Engineering (601321)1 Introduction Complex and large SW. SW crises Expensive HW. Custom SW. Batch execution.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Processes.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 3 Slide 1 Software Processes l Coherent sets of activities for specifying, designing,
CEN rd Lecture CEN 4021 Software Engineering II Instructor: Masoud Sadjadi Phases of Software.
Certification and Accreditation CS Phase-1: Definition Atif Sultanuddin Raja Chawat Raja Chawat.
Lecture 3 Software Engineering Models (Cont.)
Management & Development of Complex Projects Course Code MS Project Management Perform Qualitative Risk Analysis Lecture # 25.
Slide 1V&V 10/2002 Software Quality Assurance Dr. Linda H. Rosenberg Assistant Director For Information Sciences Goddard Space Flight Center, NASA
Lecture 7 Risk Analysis CSCI – 3350 Software Engineering II Fall 2014 Bill Pine.
Systems Analysis and Design in a Changing World, Fourth Edition
Search Engine Optimization © HiTech Institute. All rights reserved. Slide 1 What is Solution Assessment & Validation?
SOFTWARE PROJECT MANAGEMENT
SecSDLC Chapter 2.
Kathy Corbiere Service Delivery and Performance Commission
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 4 Slide 1 Software Processes.
1 Project Management C53PM Session 4 Russell Taylor Staff Work-base – 1 st Floor
Ensuring the Safety of Future Developments
Software Engineering, 8th edition. Chapter 4 1 Courtesy: ©Ian Sommerville 2006 FEB 13 th, 2009 Lecture # 5 Software Processes.
SwCDR (Peer) Review 1 UCB MAVEN Particles and Fields Flight Software Critical Design Review Peter R. Harvey.
Failure Modes and Effects Analysis (FMEA)
Failure Modes, Effects and Criticality Analysis
Advanced Software Engineering Dr. Cheng
Strategic Information Systems Planning
Chapter 6: Database Project Management
Presented to the NASA OSMA SAS ‘01
Software Processes (a)
Software Independent Verification and Validation (IV&V)
Air Carrier Continuing Analysis and Surveillance System (CASS)
Engineering Processes
Introduction to Software Testing
Engineering Processes
Presentation transcript:

SQM - 1DCS - ANULECTURE Software Quality Management Software Quality Management Processes V & V of Critical Software & Systems Ian Hirst

SQM - 2DCS - ANULECTURE Agenda review review risk based V & V risk based V & V software & systems criticality software & systems criticality criticality analysis & risk assessment (CARA) method criticality analysis & risk assessment (CARA) method impact & risk driver categories & values impact & risk driver categories & values CARA steps CARA steps CARA implementation recommendations CARA implementation recommendations CARA benefits CARA benefits

SQM - 3DCS - ANULECTURE Review many software projects fail to meet objectives many software projects fail to meet objectives lack of objective quality evidence is common lack of objective quality evidence is common a complex solution requires significant test planning and management effort and a complex set of testing activities a complex solution requires significant test planning and management effort and a complex set of testing activities complete testing is usually not possible complete testing is usually not possible testing should be focused on assisting with evaluation of the success of the project and the ‘quality’ of the delivered solution…. testing should be focused on assisting with evaluation of the success of the project and the ‘quality’ of the delivered solution….

SQM - 4DCS - ANULECTURE Risk based V & V Risk = probability of occurrence x impact Risk = probability of occurrence x impact V & V is primarily a risk management activity V & V is primarily a risk management activity risks can be associated with both product & processes risks can be associated with both product & processes high impact elements are critical elements high impact elements are critical elements V & V is aimed at reducing or eliminating uncertainty by the provision of evidence of the capability & quality of software & systems V & V is aimed at reducing or eliminating uncertainty by the provision of evidence of the capability & quality of software & systems V & V on all software is not necessary or financially feasible V & V on all software is not necessary or financially feasible risk based V & V is a targeted activity…. risk based V & V is a targeted activity….

SQM - 5DCS - ANULECTURE Software & systems criticality Criticality is a measure of the impact of errors on: Criticality is a measure of the impact of errors on: system performance and operations system performance and operations safety / security safety / security cost & schedule cost & schedule Risk is a measure of the likelihood of errors based on: Risk is a measure of the likelihood of errors based on: complexity complexity maturity of technology maturity of technology requirements definition & stability requirements definition & stability testability testability developer experience…. developer experience….

SQM - 6DCS - ANULECTURE The Criticality Analysis & Risk Assessment (CARA) method CARA is a formalised methodology which evolved from a US Air Force V & V initiative. CARA is a formalised methodology which evolved from a US Air Force V & V initiative. CARA provides a systematic procedure for rank ordering development program elements with respect to well defined scoring factors associated with criticality & risks drivers CARA provides a systematic procedure for rank ordering development program elements with respect to well defined scoring factors associated with criticality & risks drivers CARA is a means for evaluating the risk exposure for software or systems and sizing the V & V effort CARA is a means for evaluating the risk exposure for software or systems and sizing the V & V effort CARA has been applied to space systems including : CARA has been applied to space systems including : space shuttle software and critical mission support software space shuttle software and critical mission support software space station flight software…. space station flight software….

SQM - 7DCS - ANULECTURE The criticality & risk driver categories & values error impact categories & values error impact categories & values Catastrophic (4) Catastrophic (4) Critical (3) Critical (3) Marginal (2) Marginal (2) Negligible (1) Negligible (1) risk driver categories & values risk driver categories & values Complexity (high=3, moderate=2, low =1) Complexity (high=3, moderate=2, low =1) Maturity of technology (high=3, moderate=2, low =1) Maturity of technology (high=3, moderate=2, low =1) Requirements definition & stability (high=3, moderate=2, low=1) Requirements definition & stability (high=3, moderate=2, low=1) Testability (high=3, moderate=2, low=1)…. Testability (high=3, moderate=2, low=1)…. refer tables 14.3 & 14.4

SQM - 8DCS - ANULECTURE The criticality categories & values

SQM - 9DCS - ANULECTURE The risk driver categories & values

SQM - 10DCS - ANULECTURE CARA steps Step 1: Identify software functions Step 1: Identify software functions Step 2: Establish the evaluation team Step 2: Establish the evaluation team Step 3: Develop the CARA evaluation criteria Step 3: Develop the CARA evaluation criteria Step 4: Perform criticality analysis & risk assessment Step 4: Perform criticality analysis & risk assessment Step 5: Set V&V analysis level (VAL) thresholds Step 5: Set V&V analysis level (VAL) thresholds Step 6: Estimate software size Step 6: Estimate software size Step 7: Generate V&V effort estimates Step 7: Generate V&V effort estimates Step 8: Evaluate effort estimate results Step 8: Evaluate effort estimate results Step 9: Revise V&V scope…. Step 9: Revise V&V scope….

SQM - 11DCS - ANULECTURE Step 1: Identify software functions Collect systems information (from CONOPS,specs, business cases) Collect systems information (from CONOPS,specs, business cases) Identify the required software capabilities (functions and performance) Identify the required software capabilities (functions and performance) Build a scoring matrix Build a scoring matrix A solution decomposition/ PBS based structure is useful A solution decomposition/ PBS based structure is useful Values may be assigned at any appropriate level of abstraction (requirements, requirements groups, components, sub systems, etc) Values may be assigned at any appropriate level of abstraction (requirements, requirements groups, components, sub systems, etc) Identify related systems domains / areas of specialisation…. Identify related systems domains / areas of specialisation….

SQM - 12DCS - ANULECTURE Step 2: Establish the evaluation team Engage system domain experts Engage system domain experts Engage development process experts Engage development process experts Establish management team & processes…. Establish management team & processes….

SQM - 13DCS - ANULECTURE Step 3: Develop the CARA evaluation criteria Collect evaluation criteria from similar domains Collect evaluation criteria from similar domains Develop an understanding of the mission the system is to perform Develop an understanding of the mission the system is to perform Tailor criticality evaluation criteria in terms of what is catastrophic, critical, or of moderate impact to users, customers, and acquirers of the system Tailor criticality evaluation criteria in terms of what is catastrophic, critical, or of moderate impact to users, customers, and acquirers of the system Tailor risk evaluation criteria ( by inclusion of additional drivers) Tailor risk evaluation criteria ( by inclusion of additional drivers) Identify criticality area or risk driver weightings, if necessary Identify criticality area or risk driver weightings, if necessary Review criteria with the customer…. Review criteria with the customer….

SQM - 14DCS - ANULECTURE Step 4: Perform criticality analysis & risk assessment Perform criticality analysis Perform criticality analysis consider the systems components, their interactions, failure modes & effects, and concepts of operations consider the systems components, their interactions, failure modes & effects, and concepts of operations rate the functions according to criteria and scoring rationale rate the functions according to criteria and scoring rationale Perform risk analysis Perform risk analysis review software & system development, test and verification plans review software & system development, test and verification plans review development methods, testing approach, reuse plans, organisational interfaces, integration requirements, risks & risk mitigation techniques review development methods, testing approach, reuse plans, organisational interfaces, integration requirements, risks & risk mitigation techniques rate the functions according to criteria and scoring rationale rate the functions according to criteria and scoring rationale Calculate CARA scores (n = Criticality[xW] x Risk[xW]) Calculate CARA scores (n = Criticality[xW] x Risk[xW]) Rank elements in score order…. Rank elements in score order….

SQM - 15DCS - ANULECTURE Step 5: Set V&V analysis level (VAL) thresholds Functions with higher CARA scores receive higher VALs Functions with higher CARA scores receive higher VALs Example VAL thresholds: Example VAL thresholds: CARA scoreVAL 1<CARA<2None 2<CARA<5Limited 5<CARA<8Focused 8<CARA<12Comprehensive e.g. 1: Safety impact of 4, complexity risk of 3, n = 12 (unweighted) e.g. 2: Cost impact of 1, maturity risk of 2, n = 2 (unweighted)….

SQM - 16DCS - ANULECTURE Step 6: Estimate software size This step can be performed anytime before step 7 This step can be performed anytime before step 7 The size measurement is the V & V workload: The size measurement is the V & V workload: V&V work = f (no. of requirements, external interfaces, output products) An alternative size measure is the developer software size estimate (e.g. SLOC) An alternative size measure is the developer software size estimate (e.g. SLOC)

SQM - 17DCS - ANULECTURE Step 7: Generate V & V effort estimates (including an independent V & V effort estimate) Apply V&V productivity factors to size estimates (these may vary according to VALs, software complexity & size, development methods, development types (initial production, block update), domains, developer maturity and experience, V&V agent experience) Apply V&V productivity factors to size estimates (these may vary according to VALs, software complexity & size, development methods, development types (initial production, block update), domains, developer maturity and experience, V&V agent experience) Apply program project management (schedule and effort estimation) standards & conventions …. Apply program project management (schedule and effort estimation) standards & conventions ….

SQM - 18DCS - ANULECTURE Step 8: Evaluate effort estimate results Review the results with the customer Review the results with the customer If the prescribed VALs for the software functions and associated costs are acceptable, generate the critical functions list (this defines the V&V scope and priorities) …. If the prescribed VALs for the software functions and associated costs are acceptable, generate the critical functions list (this defines the V&V scope and priorities) ….

SQM - 19DCS - ANULECTURE Step 9: Revise V&V scope If the results are not acceptable, use the independent V&V estimate to re-scope the effort If the results are not acceptable, use the independent V&V estimate to re-scope the effort VAL threshold adjustments may be used to aid breadth v. depth tradeoffs VAL threshold adjustments may be used to aid breadth v. depth tradeoffs VAL selective exceptions/adjustments may be used e.g. for safety critical functions with a score of 3 or more - apply focused V&V (instead of limited).... VAL selective exceptions/adjustments may be used e.g. for safety critical functions with a score of 3 or more - apply focused V&V (instead of limited)....

SQM - 20DCS - ANULECTURE CARA implementation recommendations CARA scoring must be done by domain experts CARA scoring must be done by domain experts CARA scoring must be done in a peer review environment CARA scoring must be done in a peer review environment Project management should participate in scoring activities Project management should participate in scoring activities CARA training should be done with all personnel CARA training should be done with all personnel Automation tools are necessary for large projects Automation tools are necessary for large projects Capturing the scoring rationale is important Capturing the scoring rationale is important CARA should be repeated at least once every major development milestone…. CARA should be repeated at least once every major development milestone….

SQM - 21DCS - ANULECTURE CARA benefits The results can be used to support V&V planning & management : The results can be used to support V&V planning & management : assessment of overall and relative risks assessment of overall and relative risks allocation of fixed resources across a set of V&V objectives and tasks allocation of fixed resources across a set of V&V objectives and tasks assessment of the need for future V&V resources assessment of the need for future V&V resources establishing V&V importance levels / focus points/ priorities establishing V&V importance levels / focus points/ priorities prioritisation of items for work sequencing prioritisation of items for work sequencing CARA establishes a structured approach to V&V which increased customer visibility into risk, risk mitigation and V&V activities…. CARA establishes a structured approach to V&V which increased customer visibility into risk, risk mitigation and V&V activities….

SQM - 22DCS - ANULECTURE Review V & V is primarily a risk management activity V & V is primarily a risk management activity V & V on all software is not necessary or financially feasible V & V on all software is not necessary or financially feasible Risk based V & V is a targeted activity Risk based V & V is a targeted activity The criticality analysis & risk assessment (CARA) method may be used to analyse, plan and justify a structured risk based V & V program The criticality analysis & risk assessment (CARA) method may be used to analyse, plan and justify a structured risk based V & V program

SQM - 23DCS - ANULECTURE References Sommerville edition 7: chapter 20: Critical Systems Development chapter 24: Critical Systems Validation handout: Determining the Required Level of IV&V Program [Boughton] p , 2001, IEEE Computer Society Marvin V. Zelkowitz and Iona Rus, Understanding IV & V in a safety critical and complex evolutionary environment: the NASA space shuttle program, ICSE 23 (23 rd International Conference on Software Engineering) p , 2001, IEEE Computer Society