1. 1 Software Quality Assurance (Lecture 14) Dr. R. Mall

2. 2 Organization of this Lecture: zIntroduction Quality Engineering. zQuality control and Quality Assurance zISO 9000 zSEI CMM zSummary

3. 3 Introduction zTraditional definition of quality: yfitness of purpose, xa quality product does exactly what the users want it to do.

4. 4 Fitness of purpose zFor software products, yfitness of purpose: xsatisfaction of the requirements specified in SRS document.

5. 5 Fitness of purpose zA satisfactory definition of quality for many products: ya car, a table fan, a food mixer, microwave oven, etc. zBut, not satisfactory for software products.

6. 6 Introduction zConsider a software product: yfunctionally correct, xi.e. performs all functions as specified in the SRS document, ybut has an almost unusable user interface. xcannot be considered as a quality product.

7. 7 Introduction zAnother example: ya product which does everything that users want. ybut has an almost incomprehensible and unmaintainable code.

8. 8 Modern view of quality zAssociates several quality factors with a software product : yCorrectness yReliability yEfficiency (includes efficiency of resource utilization) yPortability yUsability yReusability yMaintainability

9. 9 Correctness zA software product is correct, yif different requirements as specified in the SRS document have been correctly implemented. yAccuracy of results.

10. 10 Portability zA software product is said to be portable, yif it can be easily made to work in different operating systems, yin different machines, ywith other software products, etc.

11. 11 Reusability zA software product has good reusability, yif different modules of the product can easily be reused to develop new products.

12. 12 Usability zA software product has good usability, yif different categories of users (i.e. both expert and novice users) can easily invoke the functions of the product.

13. 13 Maintainability zA software product is maintainable, yif errors can be easily corrected as and when they show up, ynew functions can be easily added to the product, yfunctionalities of the product can be easily modified, etc.

14. 14 Software Quality Management System zQuality management system (or quality system): yprincipal methodology used by organizations to ensure that the products have desired quality.

15. 15 Quality system zA quality system consists of the following: yManagerial Structure yIndividual Responsibilities. zResponsibility of the organization as a whole.

16. 16 Quality system zEvery quality conscious organization has an independent quality department: yperforms several quality system activities. yneeds support of top management. yWithout support at a high level in a company, xmany employees may not take the quality system seriously.

17. 17 Quality System Activities: zAuditing of projects zDevelopment of: ystandards, procedures, and guidelines, etc. zProduction of reports for the top management ysummarizing the effectiveness of the quality system in the organization. z Review of the quality system itself.

18. 18 Quality system zA good quality system must be well documented. yWithout a properly documented quality system, xapplication of quality procedures become ad hoc, xresults in large variations in the quality of the products delivered.

19. 19 Quality system zAn undocumented quality system: ysends clear messages to the staff about the attitude of the organization towards quality assurance. zInternational standards such as ISO 9000 provide: y guidance on how to organize a quality system.

20. 20 Evolution of Quality Systems zQuality systems have evolved: yover the last five decades. zPrior to World War II, yway to produce quality products: xinspect the finished products xeliminate defective products.

21. 21 Evolution of Quality Systems zSince that time, yquality systems of organizations have undergone xfour stages of evolution.

22. 22 Evolution of Quality Systems

23. 23 Evolution of Quality Systems zInitial product inspection method : ygave way to quality control (QC). zQuality control: ynot only detect the defective products and eliminate them ybut also determine the causes behind the defects.

24. 24 Quality control (QC) zQuality control aims at correcting the causes of errors: ynot just rejecting defective products. zStatistical quality control yquality of the output of the process is inferred using statistical methods yin stead of inspection or testing of all products

25. 25 Quality control (QC) zThe next breakthrough, ydevelopment of quality assurance principles

26. 26 Quality assurance zBasic premise of modern quality assurance: yif an organization's processes are good and are followed rigorously, xthe products are bound to be of good quality.

27. 27 Quality assurance zAll modern quality paradigms include: yguidance for recognizing, defining, analyzing, and improving the production process.

28. 28 Total quality management (TQM) zAdvocates: ycontinuous process improvements through process measurements.

29. 29 Business Process reengineering zA term related to TQM. zProcess reengineering goes a step further than quality assurance: yaims at continuous process improvement.

30. 30 Business Process reengineering  Our focus is reengineering of the software process.  Whereas BPR aims at reengineering the way business is carried out in any organization  not just software development organizations.

31. 31 Total quality management (TQM) zTQM goes beyond documenting processes yoptimizes them through redesign. zOver the years the quality paradigm has shifted: yfrom product assurance to process assurance.

32. 32 ISO 9000 zISO (international Standards Organization): ya consortium of 63 countries established to formulate and foster standardization. zISO published its 9000 series of standards in 1987.

33. 33 What is ISO 9000 Certification? zISO 9000 certification: yserves as a reference for contract between independent parties. zThe ISO 9000 standard: yspecifies guidelines for maintaining a quality system.

34. 34 What is ISO 9000 Certification? zISO 9000 specifies: yguidelines for repeatable and high quality product development. yAlso addresses organizational aspects xresponsibilities, reporting, procedures, processes, and resources for implementing quality management.

35. 35 ISO 9000 zA set of guidelines for the production process. ynot directly concerned about the product it self. ya series of three standards: xISO 9001, ISO 9002, and ISO 9003.

36. 36 ISO 9000 zBased on the premise: yif a proper process is followed for production: xgood quality products are bound to follow.

37. 37 ISO 9001: zApplies to: yorganizations engaged in design, development, production, and servicing of goods. yapplicable to most software development organizations.

38. 38 ISO 9002: zISO 9002 applies to: yorganizations who do not design products: xbut are only involved in production. zExamples of this category of industries: ysteel or car manufacturing industries y buy the product and plant designs from external sources: xonly manufacture products. ynot applicable to software development organizations.

39. 39 ISO 9003 zISO 9003 applies to: yorganizations involved only in installation and testing of the products.

40. 40 ISO 9000 for Software Industry zISO 9000 is a generic standard: yapplicable to many industries, xstarting from a steel manufacturing industry to a service rendering company. zMany clauses of ISO 9000 documents: yuse generic terminologies yvery difficult to interpret them in the context of software organizations.

41. 41 Software vs. other industries zVery difficult to interpret many clauses for software industry: ysoftware development is radically different from development of other products.

42. 42 Software vs. other industries zS oftware is intangible ytherefore difficult to control. xIt is difficult to control anything that we cannot see and feel. yIn contrast, in a car manufacturing unit: xwe can see a product being developed through stages such as fitting engine, fitting doors, etc. xone can accurately tell about the status of the product at any time. ySoftware project management is an altogether different ball game.

43. 43 Software vs. other industries zDuring software development: ythe only raw material consumed is data. zFor any other product development: yLot of raw materials consumed xe.g. Steel industry consumes large volumes of iron ore, coal, limestone, etc. zISO 9000 standards have many clauses corresponding to raw material control. xnot relevant to software organizations.

44. 44 Software vs. other industries zRadical differences exist between software and other product development, ydifficult to interpret various clauses of the original ISO standard in the context of software industry.

45. 45 ISO 9000 Part-3 zISO released a separate document called ISO 9000 part-3 in 1991 yto help interpret the ISO standard for software industry. zAt present, yofficial guidance is inadequate

46. 46 Why Get ISO 9000 Certification? zSeveral benefits: yConfidence of customers in an organization increases xif organization qualified for ISO 9001 certification. xThis is especially true in the international market.

47. 47 Why Get ISO 9000 Certification? zMany international software development contracts insist: ydevelopment organization to have ISO 9000 certification.

48. 48 Why Get ISO 9000 Certification? zRequires: ya well-documented software production process to be in place. ycontributes to repeatable and higher quality software. zMakes development process: yfocussed, efficient, and cost-effective

49. 49 Why Get ISO 9000 Certification? zPoints out the weakness of an organizations: yrecommends remedial action. zSets the basic framework: yfor development of an optimal process and TQM.

50. 50 How to Get ISO 9000 Certification? zAn organization intending to obtain ISO 9000 certification: yapplies to a ISO 9000 registrar for registration. zISO 9000 registration process consists of several stages.

51. 51 How to Get ISO 9000 Certification? zApplication stage: yApplies to a registrar for registration. zPre-assessment: ythe registrar makes a rough assessment of the organization.

52. 52 How to Get ISO 9000 Certification? zDocument review and adequacy audit: yprocess and quality-related documents. ythe registrar reviews the documents ymakes suggestions for improvements.

53. 53 How to Get ISO 9000 Certification?  Compliance audit: the registrar checks  whether the suggestions made by it during review have been complied.

54. 54 How to Get ISO 9000 Certification? zRegistration: yThe registrar awards ISO 9000 certificate after successful completions of all previous phases. zContinued surveillance: y The registrar continues monitoring the organization periodically.

55. 55 ISO 9000 Certification zAn ISO certified organization ycan use the certificate for corporate advertizements ycannot use the certificate to advertize products. xISO 9000 certifies organization's process xnot any product of the organization. yAn organization using ISO certificate for product advertizements: xrisks withdrawal of the certificate.

56. 56 Summary of ISO 9001 Requirements zManagement responsibility(4.1): yManagement must have an effective quality policy. yThe responsibility and authority of all those whose work affects quality: x must be defined and documented.

57. 57 Management responsibility(4.1) zResponsibility of the quality system. yindependent of the development process, ycan work in an unbiased manner. zThe effectiveness of the quality system: ymust be periodically by audited.

58. 58 Quality system (4.2) and contract reviews (4.3): zA quality system must be maintained and documented. zContract reviews (4.3): yBefore entering into a contract, an organization must review the contract xensure that it is understood, xorganization has the capability for carrying out its obligations.

59. 59 Design control (4.4): zThe design process must be properly controlled, ythis includes controlling coding also. zA good configuration control system must be in place.

60. 60 Design control (4.4): zDesign inputs must be verified as adequate. zDesign must be verified. zDesign output must be of required quality. zDesign changes must be controlled.

61. 61 Document control (4.5): zProper procedures for ydocument approval, issue and removal. zDocument changes must be controlled. yuse of some configuration management tools is necessary.

62. 62 Purchasing (4.6): zPurchased material, including bought-in software: ymust be checked for conforming to requirements.

63. 63 Purchaser Supplied Products (4.7): zMaterial supplied by a purchaser, y for example, xclient-provided software must be properly managed and checked.

64. 64 Product Identification (4.8): zThe product must be identifiable at all stages of the process. yIn software development context this means configuration management.

65. 65 Process Control (4.9) : zThe development must be properly managed. zQuality requirements must be identified in a quality plan.

66. 66 Inspection and Testing (4.10) : zIn software terms this requires effective testing i.e., yunit testing, integration testing and system testing. zTest records must be maintained.

67. 67 Inspection, measuring and test equipment(4.11): zIf integration, measuring, and test equipments are used, ymust be properly maintained and calibrated.

68. 68 Control of nonconforming product (4.13) : zIn software terms, ykeeping untested or faulty software out of released product, xor other places whether it might cause damage.

69. 69 Corrective Action (4.14) : zThis is both about correcting errors when found, yinvestigating why they occurred yimproving the process to prevent further occurrences. zIf an error reoccurs despite the quality system, ythe system needs improvement.

70. 70 Handling (4.15) and Quality audits (4.17): zHandling (4.15) Deals with: ystorage, packing, and delivery of the software product. zQuality Audits (4.17) : yquality system audit must be carried out to ensure its effectiveness.

71. 71 Training (4.18) : zTraining needs must be identified and met. zMost items of ISO standard yare largely common sense.

72. 72 Salient features of ISO 9001 requirements: zAll documents concerned with the development of a software product yshould be properly managed, authorized, and controlled. zProper plans should be prepared yprogress against these plans should be monitored.

73. 73 Salient features of ISO 9001 requirements: zImportant documents independently checked and reviewed: y for effectiveness and correctness. zThe product should be tested : yagainst specification. zSeveral organizational aspects: ye.g., management reporting of the quality team.

74. 74 Shortcomings of ISO 9001 Certification (1) zISO 9000 requires a production process to be adhered to: ybut does not guarantee the process to be of high quality. yDoes not give any guideline for defining an appropriate process.

75. 75 Shortcomings of ISO 9001 Certification (2) zISO 9000 certification process ynot fool-proof yno international accredition agency exists. ylikely variations in the norms of awarding certificates: xamong different accredition agencies and among the registrars.

76. 76 Shortcomings of ISO 9001 Certification (3) zOrganizations qualifying for ISO 9001 certification: ytend to downplay domain expertise. ytend to believe that since a good process is in place, xany engineer is as effective as any other engineer in doing any particular activity relating to software development.

77. 77 Shortcomings of ISO 9001 Certification (4) zIn manufacturing industry yclear link between process quality and product quality yonce a process is calibrated: xcan be run again and again producing quality goods zSoftware development is a creative process: yindividual skills and experience is significant

78. 78 Shortcomings of ISO 9001 Certification (5) zMany areas of software development are very specialized: yspecial expertize and experience (domain expertize) required. zISO 9001 ydoes not automatically lead to continuous process improvement, ydoes not automatically lead to TQM.

79. 79 Shortcomings of ISO 9001 Certification (6) zISO 9001 addresses mostly management aspects. zTechniques specific to software development have been ignored yConfiguration management yReviews yRelease builds yProblem Notification system yIntranets

80. 80 SEI Capability Maturity Model zDeveloped by Software Engineering Institute (SEI) of the Carnegie Mellon University, USA: yto assist the U.S. Department of Defense (DoD) in software acquisition. yThe rationale was to include: xlikely contractor performance as a factor in contract awards.

81. 81 SEI Capability Maturity Model zMajor DoD contractors began CMM- based process improvement initiatives: yas they vied for DoD contracts. zSEI CMM helped organizations: yImprove quality of software they developed yRealize adoption of SEI CMM model had significant business benefits. zOther organizations adopted CMM.

82. 82 SEI Capability Maturity Model zIn simple words, yCMM is a model for apprising the software process maturity of a contractor into different levels. yCan be used to predict the most likely outcome to be expected from the next project that the organization undertakes.

83. 83 SEI Capability Maturity Model zCan be used in two ways: yCapability evaluation ySoftware process assessment.

84. 84 Capability Evaluation zProvides a way to assess the software process capability of an organization yHelps in selecting a contractor yIndicates the likely contractor performance

85. 85 Software Process Assessment zUsed by an organization to assess its current process: ySuggests ways to improve the process capability. yThis type of assessment is for purely internal use.

86. 86 SEI Capability Maturity Model  The SEI CMM classifies software development industries into:  Five maturity levels.  Stages are ordered so that improvements at one stage provide foundations for the next  Based on the pioneering work of Philip Crosby

87. 87 SEI Capability Maturity Model Initial (1) Repeatable (2) Defined (3) Managed (4) Optimizing (5)

88. 88 Level 1: (Initial) zOrganization operates ywithout any formalized process or project plans zAn organization at this level is characterized by yad hoc and often chaotic activities.

89. 89 Level 1: (Initial) zSoftware production processes are not defined, ydifferent engineers follow their own process ydevelopment efforts become chaotic. yThe success of projects depend on individual efforts and heroics.

90. 90 Level 2: (Repeatable) zBasic project management practices ytracking cost, schedule, and functionality are followed. zSize and cost estimation techniques yfunction point analysis, COCOMO, etc. used. zProduction process is ad hoc ynot formally defined yalso not documented.

91. 91 Level 2: (Repeatable) zProcess used for different projects might vary between projects: yearlier success on projects with similar applications can be repeated. yOpportunity to repeat process exist when a company produces a family of products.

92. 92 Level 3: (Defined) zManagement and development activities: ydefined and documented. yCommon organization-wide understanding of activities, roles, and responsibilities.

93. 93 Level 3: (Defined) zThe process though defined, yprocess and product qualities are not measured. zISO 9001 aims at achieving this level.

94. 94 Level 4: (Managed) zQuantitative quality goals for products are set. zSoftware process and product quality are measured: yThe measured values are used to control the product quality. yResults of measurement used to evaluate project performance xrather than improve process.

95. 95 Level 4: (Managed) zOrganization sets quantitative quality goals zWorld-wide about 100 organizations assessed at this level.

96. 96 Level 5: (Optimizing) zStatistics collected from process and product measurements are analyzed: ycontinuous process improvement based on the measurements. xKnown types of defects are prevented from recurring by tuning the process xlessons learned from specific projects incorporated into the process

97. 97 Level 5: (Optimizing) zIdentify best software engineering practices and innovations: y tools, methods, or process are identified ytransferred throughout the organization zWorld-wide about 50 organizations have been assessed at this level.

98. 98 Key Process Areas zEach level is associated with a key process area (KPA) identifies ywhere an organization at the previous level must focus to reach this level

99. 99 Level 2 KPAs zSoftware project planning ySize, cost, schedule. yproject monitoring zConfiguration management zSubcontract management

100. 100 Level 3 KPAs zProcess definition and documentation zReviews zTraining program

101. 101 Level 4 KPAs zQuantitative measurements zProcess management

102. 102 Level 5 KPAs zDefect prevention zTechnology change management zProcess change management

103. 103 Comparison between ISO 9001 and SEI CMM zISO 9001 awarded by an international standards body ycan be quoted in official documents and communications zSEI CMM assessment is purely for internal use.

104. 104 Comparison between ISO 9001 and SEI CMM zSEI CMM was developed specifically for software industry: yaddresses many issues specific to software industry. zSEI goes beyond quality assurance yaims for TQM yISO 9001 correspond to SEI level 3.

105. 105 Comparison between ISO 9001 and SEI CMM zSEI CMM provides a list of key areas yon which to focus to take an organization from one level to the other zProvides a way for gradual quality improvements over several stages. ye.g trying to implement a defined process before a repeatable process: x counterproductive as managers are overwhelmed by schedule and budget pressure.

106. 106 Remarks on Quality Model Usage zHighly systematic and measured approach to software development process suits certain circumstances ynegotiated software, safety-critical software, etc zWhat about small organizations? xTypically handle applications such as internet, e-comm. xwithout an established product range, xwithout revenue base, experience on past projects, etc. xCMM may be incompatible

107. 107 Small Organizations zSmall organizations tend to believe: yWe are all competent people hired to do a job, we can’t afford training yWe all communicate with one another xOsmosis works because we are so close yWe are all heroes xWe do what needs to be done xTherefore rules do not apply to us

108. 108 Small Organizations zOften have problems: yUndocumented requirements yInexperienced managers yDocumenting the product yResource allocation yTraining yPeer reviews

109. 109 Small Organizations zA two week CMM-based appraisal is probably excessive: zSmall organizations need to operate more efficiently at lower levels of maturity yMust first fluorish if eventually they are to mature

110. 110 Personal Software Process (PSP) zBased on the work of Humphrey zPSP is a scaled down version of industrial software process ysuitable for individual use zEven CMM assumes that engineers use effective personal practices

111. 111 Personal Software Process (PSP) zA process is the set of steps for doing a job zThe quality and productivity of an engineer ylargely determined by his process zPSP is framework that yhelps software engineers to measure and improve the way they work.

112. 112 Personal Software Process (PSP) zHelps developing personal skills and methods yEstimating and planning method yShows how to track performance against plans yProvides a defined process xcan be fine tuned by individuals xRecognizes that a process for individual use is different from that necessary for a team project.

113. 113 Time Management zTrack the way you spend time yBoring activities seem longer then actual yInteresting activities seem short zRecord time for yDesigning yWriting code yCompiling yTesting

114. 114 Personal Software Process (PSP) Planning Design Code Compile Test Postmortem Logs Project plan summary

115. 115 PSP-Planning zProblem definition zEstimate max, min, and total LOC zDetermine minutes/LOC zCalculate max,min, and total development times zEnter the plan data in project plan summary form zrecord the planned time in Log

116. 116 PSP-Design zDesign the program zRecord the design in specified format zRecord the Design time in time recording log

117. 117 PSP-Code zImplement the design zUse a standard format for code text zRecord the coding time in time recording log

118. 118 PSP-Compile zCompile the program zFix all the defects zRecord compile time in time recording log

119. 119 PSP-Test/Postmortem zTest yTest the program yFix all the defects found yRecord testing time in time recording log zPostmortem yComplete project plan summary form with actual time and size data yRecord postmortem time in time record

120. 120 Personal Software Process (PSP) PSP 0 PSP 1 PSP 2 PSP 3  Personal measurement  Basic size measures  Personal planning  Time and schedule  Personal quality management  Design and code reviews  Personal process evolution

121. 121 Six Sigma zSix sigma is a quantitative approach to eliminate defects yApplicable to all types of industry - from manufacturing, product development, to service zThe statistical representation of Six Sigma quantitatively describes yhow a process is performing

122. 122 Six Sigma zTo achieve six sigma ya process must not produce more than 3.4 defects per million opportunities. y5 Sigma -> 230 defects per million y4 Sigma -> 6210 defects per million zSix sigma methodologies yDMAIC (Define, Measure, Analyze, Improve, Control) yDMADV: (Define, Measure, Analyze, Design, Verify)

123. 123 Six Sigma Methodologies zThe methodologies are implemented by Green belt and Black belt workers ySupervised by Master black belt worker zPareto Chart: ySimple bar chart to represent defect data yIdentify the problems that occurs with greatest frequency xor incur the highest cost

124. 124 Summary zEvolution of quality system: yproduct inspection yquality control yquality assurance ytotal quality management (TQM) zQuality paradigm change: yfrom product to process

125. 125 Summary zISO 9000: ybasic premise: xif a good process is followed xgood products are bound to follow yprovides guidelines for establishing a quality system.

126. 126 Summary zISO 9000 yseries of three standards x9001, 9002, and 9003 y9001 is applicable to software industry

127. 127 Summary zSEI CMM ydeveloped specially for software industry yclassifies software organizations into five categories. xAccording to the maturity of their development process.

128. 128 Current Trends zMany organizations have already tuned their process for yBudget, ySchedule, and yQuality product. zCompetition is challenging them to: yReduce time for delivery yAdopt Six-Sigma methodology