Copyright , Dennis J. Frailey CSE7315 – Software Project Management CSE7315 M34 - Version 9.01 SMU CSE 7315 Planning and Managing a Software Project Module 34 Software Quality Assurance

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Objectives of This Module  To examine the techniques of software quality assurance  To look at some techniques of software quality engineering –Notably, cost of quality analysis

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Software Quality Engineering and Assurance References to texts: Humphrey Futrell  Managing Quality16 30  Engineering Quality9,10 30  Software Testing11 23  Other Relevant Material 8 20

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Evolution of Software Quality Quality Control Quality Assurance Quality Engineering This Module Introduced in This Module

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Software Quality Assurance Purposes: –To provide management and developers with visibility into the process being followed and the products being built  So they can manage the outcome –To provide a more effective quality control mechanism

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Software Quality Assurance Typical Practices: –Inspections –Reviews –Audits –Communication –Measurements –Independent confirmation of compliance  Standards, requirements, procedures

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Quality Assurance Goal: To assure that quality is achieved -- Add quality during development -- Improve processes, standards, and up- front evaluations of the product as it is being developed Quality Control Quality Assurance Quality Engineering

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Quality Assurance Looks at the Entire Process Requirements DevelopmentQC Inspection Pass Fail Standards of Quality Process and Design Standards

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Each Step of the Process Has an Effect on The Final Defect Rate Process Step I = Defects Input O = Defects Output F = Defects Found and Fixed C = Defects Created O = I + C - F

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Typical Responsibilities of QA  Review all development plans for completeness  Participate as inspection moderators in design and code inspections  Make sure the tests are appropriate and are run correctly –Note – true quality assurance does not develop or run tests – it assures they are done correctly  Review a significant sample of all test results to determine adherence to standards  Periodically audit SCM performance to determine adherence to standards

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At Each Step, QA must...  Inspect the product for –Conformance with standards –Satisfaction of requirements  Evaluate the process for –Conformance with standards –Opportunities to improve –Process defects that may cause problems later on  Evaluate the support processes as well

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At Each Step, QA must...  Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives  Trace back to prior steps when defects are found In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each new project

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At the Requirements Analysis Step...  Trace software requirements or specifications to original system or customer requirements  Inspect or evaluate software requirements against standards to see that they are -- complete, -- correct, -- consistent, -- testable, and -- understandable  “Clean Room” technique takes this farther PROGRAM SAMPLE (IN1,IN2) INTEGAR IN1,IN2,A,B,C; FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIF ENDDO ** THE NEXT PART WILL SORT DO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i) ENDLOOP CALL CHECKORDER(ARRAY) ** NEXT WE DO THE i/o CALL PRINT(ARRAY) CALL DEBUG (ARRAY, I) CASE I OF CALL SUB1

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version “Clean Room” Software Development Technique  Motive: don’t let the bugs get in  Method: filter them out from the beginning -- Define the requirements using a formal notation –Reduces ambiguity –Some consistency/correctness issues can be checked automatically -- Carry out rigid inspections  Benefits: fewer problems later in the development process (e.g., reduced testing).

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At the Design Step...  Trace the design to the requirements specification (plus “derived requirements”)  Evaluate the design against standards -- complexity -- cohesion -- understandability -- etc.  Evaluation of design effectiveness, correctness, consistency, completeness  Evaluate plans for testing and test cases

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At the Coding Step...  Trace code to design specifications  Evaluate code against coding standards (example: give code to a new maintainer and see if she or he can understand it)  Code walkthroughs or inspections to check on coding mistakes  Test coverage analysis -- Do the tests address all requirements? (black box tests) -- Do the tests adequately evaluate the design and its implementation? (glass box tests)

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version At the Module Testing Step...  Test procedures and code analysis -- Evaluate test code like any other code -- Are the procedures documented? -- Are the expected results documented?  Test results analysis -- Were the tests run? -- Were the results correct?

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Two Philosophies of Testing Code Test a Little Review Thorough Test CodeReview Thorough Test

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version During Integration...  Make sure that regression tests are performed -- Retest all modules potentially affected by any change  Do independent tests -- Someone other than the person who developed the software  Formal qualification tests (acceptance tests)

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Planning for QA  Each development project should have a documented Software Quality Assurance Plan (SQAP) –May be part of SW Development Plan  The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure  There is an IEEE standard for SQAPs

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Pitfalls in Implementing QA  Lack of sufficiently skilled staff in QA –Software professionals are typically more interested in development jobs than QA  QA management cannot effectively negotiate with the development team  Senior management commitment may fade under schedule pressures –The developers stop caring about QA & action revolves around insignificant issues

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Pitfalls in Implementing QA (continued)  QA organization operates without suitably documented and approved development standards and procedures  Development team does not produce verifiable quality plans –Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Pitfalls in Implementing QA (continued)  Disputes over who has responsibility for product quality and for performing QA functions –The software developers and managers are responsible for product quality –Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Reporting Chain for QA Staff  Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team –This improves their objectivity –And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences Vice President Quality Mgr Project Mgr Software Mgr Software Engineer Quality Engineer

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Benefits Achieved by Quality Assurance over Quality Control  Significant insight is gained into the process that can be used to gain long term benefits –Development can focus on specific weak points in the quality –Process improvements can be identified –Steps that are difficult to perform can be automated

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Problems with Quality Assurance  Can still be adversarial (although perhaps less than with quality control)  Motivation to fix the process is still weak  Can be more costly than quality control  It can be hard to show benefits until long after the product has been shipped

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Quality Engineering Goal: Build quality in as part of the software engineering process A philosophical change that relies on the professional pride of the software engineer  Engineering staff define and execute the quality assurance tasks  Team approach to quality, with rewards based on quality  Quality professionals are more like coaches and educators than evaluators Quality Control Quality Assurance Quality Engineering

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Quality Engineering Focuses on People As Well As Process  Finding errors is good -- it keeps them from leaking through to the customer  Problems result in process changes, not punishment of people  Everyone appreciates that a competitive process is the way to remain a competitor  Measurements are used so that decisions are based on fact (in addition to intuition)  Independent tests are welcome -- they tell you if you are as good as you think you are

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Peer Reviews Motivation: how to find problems in your work in a non-threatening way Method: evaluation of one individual’s work by others at the same organizational level (i.e., team members, not supervisors) Software DevelopersSoftware Managers

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Peer Reviews - Cost Cost: may require 10-20% of the total development time -- this must be planned into the schedule -- it must also be planned into progress measures and reward systems

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Peer Reviews - Benefits Benefits: reduced rework and faster development due to a more effective evaluation -- the peers understand the software best -- and they will not be as much of a threat when it comes to promotions and raises -- reciprocity -- you help them and they help you

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Big Problem with Peer Reviews People don’t want to attend others’ peer reviews because they are behind in their own schedules Solutions? Class discussion

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Benefits of the Quality Engineering Approach  Less adversarial  Motivation and information to improve  Flexibility to change the process in response to a problem -- you understand the problem and its cause -- you understand the consequences of a change in the process  Knowledge is the foundation of successful quality engineering

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version The Cost of Quality Engineering  Your process may be more complex, costly and time consuming - at least when you start  But you gain knowledge -- what and where to automate -- how and where to improve the process  And you gain higher quality  And you can begin to optimize the process, just as you optimize software

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Cost of Quality Analysis

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Quality (Fewer Defects; Customer Satisfaction) Productivity Cycle Time Customer Value Quality Improvements Cost Money  How can we justify them?

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Managers and Technical Staff must be convinced that  Quality problems are serious  Poor quality costs them money  Quality is worth fixing  Quality can be fixed by proper techniques In Order to Justify Quality Improvements...

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version The Fundamental Prejudice  Quality improvement is a non-value-added activity –It costs overhead resources –The benefits are not necessarily real  So we seek to avoid it Cost of Quality (COQ) Analysis is used to address this prejudice

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version The First Question This is what management and technical staff will typically ask when approached with a proposal to improve quality “What does it cost to improve quality?”

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version The Right Questions “What’s the payoff?” “What is the return on investment for quality improvements?

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Categorizing Quality-Related Costs Quality Related Costs Cost of Conformance Cost of Non- Conformance Prevention Costs Appraisal Costs Internal Failure Costs External Failure Costs

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Cost of Conformance  Things that improve quality –Prevention costs - prevent poor quality  Predictive measurements, training, root cause analysis, process improvement –Appraisal costs - detect poor quality  Tests, inspections, audits

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Cost of Non-Conformance  The price you pay when you fail to achieve quality –Internal failures  Costs incurred before product shipment –External failures  Costs incurred after product shipment

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Effects of Maturity on Costs SEI Maturity Level Cost as a Percent of Development Cost Prevention Appraisal Internal Failures External Failures Total COQ As reported by Knox (see references)

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version Module Summary  Quality Control focuses on the end product  Quality Assurance focuses on the process  Quality Engineering focuses on process and people, integrating QA with the software engineering process

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version References  Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute  Meyer, Bertrand, 1997, Object Oriented Software Construction  Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview.  Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN

Copyright , Dennis J. Frailey CSE7315- Software Project Management CSE7315 M34 - Version END OF MODULE 34