1. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 SMU CSE 8314 Software Measurement and Quality Engineering Module 07 Attributes of a Quality Product, Part 2

2. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 2 Attributes of a Quality Product  Reliability  Maintainability  Verification  Validation  Testing and Evaluation  Safety  Supportability

3. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 Attributes of a Quality Product Safety

4. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 4 Safety A product is safe if it will not cause harm  This may be hard to test  Human factors often play a big part  Software safety must be evaluated from the perspective of the system that the software is a part of –e.g. safety issues in a word processor vs. an aircraft system

5. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 5 The Safety Process Determine System Hazards (Hazard Analysis) Identify Safety Critical Components Determine Probability or Level of Control (for each component) Take Appropriate Steps During Development & Testing System Analysis Software Analysis

6. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 6 A Hazard Is … … anything that might cause harm to a person, property, or the environment  Note that often the exact definition is determined by the customer. Determine System Hazards (Hazard Analysis)

7. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 7 Hazards are Evaluated for Severity Typical Hazard Severities CategoryDescriptionSeverity of Hazard Effect ICatastrophicDeath, Loss of System IICriticalSevere Injury, Illness, Major System or Environmental Damage IIIMarginalMinor Injury, Illness, or System or Environmental Damage IVNegligibleLess than Minor Injury, Illness, or System or Environmental Damage Severity Category is Used to Select Appropriate Mitigation Actions

8. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 8 Hazards are Evaluated for Probability Typical Hazard Probabilities LevelDescription Probability AFrequentWill Occur1 in 100 BProbableLikely to Occur1 in 1000 COccasionalUnlikely but Possible 1 in 10,000 DRemoteVery Unlikely1 in 100,000 EImprobableCan Assume Will Not Occur 1 in 1,000,000 Probability Level is Used to Select Appropriate Mitigation Actions

9. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 9 Hazard Risk Index Combines Severity and Probability Typical Hazard Risk Index Calculation SeverityIIIIIIIV Probability CatastrophicCriticalMarginalMegligable A 1 in 100UUMN B 1 in 1000UUMN C 1 in 10000UMNN D 1 in 100000MNNN E 1 in 1000000NNNN U - UnacceptableMitigate at High Cost M – Marginal RiskMitigate at Moderate Cost N – No Significant RiskMitigate at Low Cost

10. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 10 A Safety Critical Component is … … a component that might contribute to a hazard if it fails  Note that in most hazardous systems there are many components that can contribute to a safety hazard  Some components may be software Identify Safety Critical Components

11. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 11 Level of Control  For Software, probability of occurrence is generally difficult or impossible to calculate.  Therefore, for software the probability is typically replaced by a “software level of control” scale  But a hazard risk index is still calculated Determine Level of Control (for each component)

12. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 12 Hazard Risk Index for Software Typical Software Hazard Risk Index Calculation Level of ControlIIIIIIIV Probability AutonomousInformation Only Operator Control No Involvement A 1 in 100UUM Not Safety Software B 1 in 1000UUM Not Safety Software C 1 in 10000UMN Not Safety Software D 1 in 100000MNN Not Safety Software E 1 in 1000000NNN Not Safety Software U - Unacceptable Mitigate at High Cost M – Marginal Risk Mitigate at Moderate Cost N – No Significant Risk Mitigate at Low Cost

13. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 13 Appropriate Actions  There are many views regarding what actions are appropriate for safety critical software  Most of the methods suggested do not work if you are using “off the shelf” software from some other organization (COTS)  For high hazard risk index software, COTS is usually not recommended unless it is certified in some fashion Take Appropriate Steps During Development & Testing Certified Safe Software

14. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 14 Examples of Appropriate Actions PhaseActionsUMN RequirementsSafety Requirements Documented Hazard Analysis Trace Hazards to Requirements DesignTrace Hazards to Design CodeCheck Code for Safety Issues during Code Inspections or Walkthroughs TestFunctional Testing Stress Testing Stability Testing 100% Branch Testing During All Phases Change Requests must be Evaluated for Impact on Safety Critical Software Maintain Hazard Control Records

15. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 15 Software Safety  Seldom addressed in Computer Science programs  But today, product problems are more and more being attributed to software, and more of this is likely.  This subject is covered in more depth in other SMU courses See end of module for recommended standards and guidelines for software safety.

16. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 16 Software Liability  The day may soon come when product liability is attributed to software developers. Consider the software that controls a nuclear reactor. Who will be liable if it fails and causes a major disaster?

17. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 17 Principles of Professional Software Development  These are still emerging  They are needed before software engineering can be considered a professional discipline

18. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 18 Is a Software Engineer an Engineer?  It is illegal in many states to call yourself a software engineer unless you are a registered (licensed) professional engineer  Several countries and at least one US State now license software engineers as professional engineers

19. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 19 Is a Software Engineer an Engineer?  CSAB/ABET has begun to accredit software engineering academic programs  IEEE has introduced a certification program for software developers  ASQ has a certification program for software quality engineers  ACM and IEEE have endorsed a software engineering code of ethics

20. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 20 Software Engineering Body of Knowledge  IEEE Computer Society and eleven other sponsors have developed SWEBOK - a “guide” to the software engineering body of knowledge www.swebok.org  This guide is being used to define software engineering curricula in industry and academia

21. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 21 In Short...  The time is coming when software development professionals will be expected to adhere to recognize standards, to utilize accepted practices, and to know basic facts about their discipline  The attorneys are educating themselves about these matters –Which means there will soon be legal implications for unsafe software

22. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 Attributes of a Quality Product Supportability

23. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 23 Supportability Can it be readily supported?  Can the software be updated (in the field)?  Can the software be examined to determine its release, version, contents, etc.?  Can the software be evaluated?  Can the software be tested?  etc.

24. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 24 Supportability has Design Implications  Providing access to the software –Data paths –Interfaces –Modes of operation that permit access  Providing a means of modifying –ROM vs. RAM –Memory loading & verification/validation  Organizing to facilitate upgrades –How are the components combined?

25. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 25 Design for Supportability: Background for Example  With many large software systems, you often designate individual “software items” or “products”.  Each software item is a stand-alone element that has its own price, its own part number, perhaps its own contract, and requires its own documentation, maintenance procedures, etc.

26. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 26 Background (2 of 3)  Example: on a PC, the software items might be the operating system, word processor, spreadsheet, database, and such.  Alternate example: you could “bundle” the above into one software item - but that would mean you cannot upgrade the spreadsheet without upgrading all the others.

27. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 27 Background (3 of 3)  Alternate example: you could “unbundle” the spreadsheet into the user interface, macro processor, arithmetic processor, and other parts –But this would be needlessly complex and expensive for the customer.  Selection of software items is a major decision when designing a software system.

28. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 28 Design for Supportability IFF System Example Function: respond to a signal and identify yourself as a “friend” Application: used on military vehicles in a combat zone to guard against accidental attack by “friendly fire” IFF System Identify yourself! Friend or Foe? Friend -- Here’s the Password xxxxx Applications: Airplanes -- All services, primarily USAF Ships -- US Navy Tanks, ATVs, etc. -- US Army

29. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 29 Basic Architecture of IFF System Common to All Applications Interfac e (unique to each application) Use of common elements saves money in development, testing and production

30. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 30 Basic Design of IFF Computer System RAM (fast, but loses contents when powered down) Low Speed ROM (large, low cost, slow) Note: Three separate memory systems Software is distributed as ROM memory sticks I/ O I/F CPUCPU download at power up HIGH SPEED ROM (small, costly, read-only, but retains contents)

31. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 31 Three Versions of the Product ROMROM RAMRAM ARMY VERSION AIR FORCE VERSION NAVY VERSION I/F COMMON I/F

32. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 32 How to Partition the Software? Option 1  Three Software Items:  Most of the software is common –saves money

33. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 33 Issues with Option 1  Supportability Problem: –Changing the common portion of any one version requires changing (and re-testing) the other two -- or else losing the commonality  Also: An upgrade requires changing two memory components - more costly

34. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 34 Option 2  Four software items:  Can avoid requiring separate memory chips for common and interface by having a single chip contain two software items: –Common plus selected interface

35. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 35 Issues with Option 2  Problem: –A change in the ROM part requires a change in the RAM part as well (and vice versa) –Must replace two memory components.  Also –Two software items on one memory stick causes complications in paperwork

36. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 36 Option 3  Six software items: –ROM & RAM part for each service

37. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 37 Issues with Option 3  Problem: –Not a logical approach from a software design perspective –A change to any one version requires re- testing the common part –You still have paperwork problems  But it is probably the most logical from a supportability perspective

38. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 38 Option 4  Eight Software Items:

39. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 39 Issues with Option 4  Many parts to stock and keep straight  Excessive documentation cost  Marginal if any benefit over option 3

40. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 40 An Additional Problem  Corporate and/or customer standards and policies generally dictate certain kinds of documentation, review processes, etc. for each software item.  In order to avoid excessive cost, program managers often decide arbitrarily on a small number of software items.

41. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 41 A Preferred Approach  Select the right number of software items and tailor out unnecessary documentation, reviews, etc.  But that takes knowledge, time, preparation, planning, etc. -- i.e., MATURITY

42. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 42 Summary  There are many attributes associated with product quality  Each of these attributes requires effective planning and analysis  But addressing these elements results in significant improvements in quality and savings in long term cost  These must be addressed in the context of how people function and how people fail

43. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 43 References  Department of Defense, Standard Practice for System Safety, MIL-STD-882D (2000). Available from various web sites.  Marciniak and Evans. Software Quality Assurance and Management.  NASA, Software Safety Guidebook, NASA- GB-8719.13 (2004). Available from various web sites.  Schulmeyer, G. Gordon and James McManus. Handbook of Software Quality Assurance, Second Edition. Van Nostrand Reinhold, New York, 1992. ISBN 0-442-00796-5.

44. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M07 - Version 7.09 44 END OF MODULE 07