2. Tom Gilchrist tomg@halcyon.com The Tools and Techniques of SQA SASQAG, February 17, 2000

3. SASQAG 2/17/00 tomg@halcyon.com Disclaimers The views and opinions expressed in this presentation are not necessarily those of my employer. Your results may vary. You are free to use any slide in your work, but if you do, please include my name and email.

4. SASQAG 2/17/00 tomg@halcyon.com Goals and Objectives Some Terms Peer Reviews Kickoff Meetings Audits Checklist and Worklists Sharpen Your SQA Skills

5. SASQAG 2/17/00 tomg@halcyon.com Software Quality Assurance Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Checks/Audits) Review and measure the quality of software products and processes throughout development. (Peer Reviews) Provide software project management (and other appropriate parties) with the results of reviews and process checks. Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.

6. SASQAG 2/17/00 tomg@halcyon.com Validation and Verification Validation -Are we building the right product? Verification -Are we building the product right? Static Verification (of software) -Verification of a program by examination and analysis of its program elements.

7. SASQAG 2/17/00 tomg@halcyon.com Quality In Perspective

8. SASQAG 2/17/00 tomg@halcyon.com Customer vs. End User End User: -The person who uses the software product in the value stream Customer: -Anyone who is affected by the end product or intermediate software engineering deliverables being produced (reviewed).

9. SASQAG 2/17/00 tomg@halcyon.com Healthy Customer/Supplier Relationship Implies product use by customer! Implies choice Communications centers on expectations. -How we identify expectations -How we document and communicate them -How we measure/improve them

10. SASQAG 2/17/00 tomg@halcyon.com Value vs Quality Customers are concerned with value not just quality.

11. SASQAG 2/17/00 tomg@halcyon.com Value vs Quality Customers are concerned with value not just quality. "Value is not overpaying for quality." Value is..."the right combination of product quality, fair price, and good service." * From Business Week, Nov 11, 1991

12. SASQAG 2/17/00 tomg@halcyon.com Low Quality in Software Unneeded ComplexityUnneeded Complexity ReworkRework -Wrong -Unclear -Missing -Extra

13. SASQAG 2/17/00 tomg@halcyon.com The Project Requirements Finished Product SpecsDesignCodeTest Time Project Plan

14. SASQAG 2/17/00 tomg@halcyon.com Where is rework introduced? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

15. SASQAG 2/17/00 tomg@halcyon.com Where is rework found? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

16. SASQAG 2/17/00 tomg@halcyon.com Where is rework fixed? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

17. SASQAG 2/17/00 tomg@halcyon.com The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production 40-100X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981

18. SASQAG 2/17/00 tomg@halcyon.com The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production 40-100X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981 High Visibility/Reward

19. SASQAG 2/17/00 tomg@halcyon.com The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production 40-100X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981 High Visibility/Reward Low Visibility/Reward

20. SASQAG 2/17/00 tomg@halcyon.com Error/Defect Classification Number of Errors MinorMajor"Show Stoppers" Error Severity

21. SASQAG 2/17/00 tomg@halcyon.com Error/Defect Classification Number of Errors MinorMajor"Show Stoppers" Find and remove as many Major and Show Stopper errors as early as possible. Error Severity

22. SASQAG 2/17/00 tomg@halcyon.com Terminology Major ErrorMajor Error -Something that is wrong -High degree of probability that it will cause rework -Rework later will cost much more if left allowed to migrate downstream. Minor ErrorMinor Error An issue or defect that is not major. The costs of fixing later is not significantly higher than fixing now.

23. SASQAG 2/17/00 tomg@halcyon.com Quality Metric…Rework Number of Major Errors / Unit of WorkNumber of Major Errors / Unit of Work -ME/Page -ME/Drawing Zone -ME/KLOC -ME/?

24. SASQAG 2/17/00 tomg@halcyon.com Peer Reviews

25. SASQAG 2/17/00 tomg@halcyon.com Initial Level Informal Checking Do Work Down- stream Customers Input Source Materials Errors Found Process Product Entry Criteria Loosely Defined Exit Criteria For Quality Not Defined Work Process ENTRYENTRY EXITEXIT

26. SASQAG 2/17/00 tomg@halcyon.com Initial Level Informal Checking Do Work Down- stream Customers Re-Work Input Source Materials Errors Found Process Product Entry Criteria Loosely Defined Exit Criteria For Quality Not Defined Work Process ENTRYENTRY EXITEXIT

27. SASQAG 2/17/00 tomg@halcyon.com Measure and Improve Product Quality Do Work Down- stream Customers Re- Work Errors Found Process Product Crossfunctional Checklists SW Inspection Input Source Materials Entry Criteria Loosely Defined Exit Criteria For Quality Defined ENTRYENTRY EXITEXIT Work Process

28. SASQAG 2/17/00 tomg@halcyon.com Improve Product Quality and Reduce Variation Do Work Down- stream Customers Re- Work Errors Found Process Product Crossfunctional Checklists Input Source Materials Entry Criteria Defined Exit Criteria For Quality Defined ENTRYENTRY EXITEXIT Work Process SW Inspection

29. SASQAG 2/17/00 tomg@halcyon.com To Improve Product AND Process (Entry Criteria Met) Do Work Down- stream Customers Re- Work Input Source Materials Errors Found Process Product Crossfunctional Checklists Exit Criteria Met Statistical Data Analysis and Causal Analysis P D C A Deming Cycle (Defect Prevention Process) Repeatable Work Process SW Inspection

30. SASQAG 2/17/00 tomg@halcyon.com SW Inspection Process Flow Planning Collect and Store Data Another Chunk? Verify and Approve AuthorsModeratorInspection Team Members OK Re-Work YES NO Other Inspection and CI Processes Produce Product Rework Product Release Individual Preparation Overview Meeting Error Logging Meeting 7

31. SASQAG 2/17/00 tomg@halcyon.com SDLC ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation Time

32. SASQAG 2/17/00 tomg@halcyon.com SW Inspections ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation SW Inspection

33. SASQAG 2/17/00 tomg@halcyon.com Kickoff Meeting ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting SW Inspection

34. SASQAG 2/17/00 tomg@halcyon.com Task Kickoff Meeting Identify people/organizations affected by errors in the activity Identify and document potential major errors (build/modify checklists) Confirm task/process entry conditions Identify resources Forum for learning Reduce the amount of time needed by the author(s) to produce a deliverable by giving technical and resource information to do the job.

35. SASQAG 2/17/00 tomg@halcyon.com Kickoff Meeting ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Program Element Creation Program Element Creation SW Inspection

36. SASQAG 2/17/00 tomg@halcyon.com Walkthrough ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Walkthrough Program Element Creation Program Element Creation Walkthrough 21 Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting SW Inspection

37. SASQAG 2/17/00 tomg@halcyon.com Walkthrough Process Reduce the amount of time needed for re-work by reviewing work completed to date and address questions and concerns. Led by author Discussion of deliverable style and format Examine technical alternatives Evaluate checklists, standards, and rules. Capture undocumented issues Forum for learning

38. SASQAG 2/17/00 tomg@halcyon.com Peer Review Methods Walkthrough MethodsTypical GoalsTypical Attributes Minimal overhead Developer training Quick turnaround Little/no preparation Informal process Meetings No measurement Led by the author SW Inspection Detect and report all defects efficiently and effectively. Usage of deliverable Formal process Known Coverage Rate Moderator Checklists Customer Reviewers Measurement Desk Check Minimal overhead Quick turnaround Little/no preparation Informal process No measurement No Meetings Done by the author

39. SASQAG 2/17/00 tomg@halcyon.com SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting

40. SASQAG 2/17/00 tomg@halcyon.com SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check

41. SASQAG 2/17/00 tomg@halcyon.com SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check SW Inspection

42. SASQAG 2/17/00 tomg@halcyon.com SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check SW Inspection Formal Walkthrough Formal Desk Check

43. SASQAG 2/17/00 tomg@halcyon.com Peer Reviews ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Walkthrough Desk Check Walkthrough Desk Check Program Element Creation Program Element Creation SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check Walkthrough Desk Check Walkthrough Desk Check

44. SASQAG 2/17/00 tomg@halcyon.com Audits

45. SASQAG 2/17/00 tomg@halcyon.com Audit Process Software Quality Assurance is to provide management with appropriate visibility into the process being used by the software developers and of the products being built. Determine: Conformance to policies, procedures, and standards Adequacy of policies, procedures, and standards Effectiveness and efficiency of policies, procedures, and standards Assess personnel familiarity to requirements and documentation Assure availability, use and adherence to software standards

46. SASQAG 2/17/00 tomg@halcyon.com Audit Process Un- satisfactory Report? OK NO YES Produce Product Corrective Actions Audit Kickoff Meeting Develop Audit Checklist Conduct Audit DevelopersAuditor & Project Manager Write-up Report & Findings Auditor Identify Requirements Follow-up Audit Re-Work Close Audit & File END Start Review with Manager

47. SASQAG 2/17/00 tomg@halcyon.com Checklists & Worklists

48. SASQAG 2/17/00 tomg@halcyon.com Checklists Used to identify Major Errors Usually in the form of questions When used…. -By the authors to keep from inserting rework into the work stream -By reviewers to find and report as many Major Errors as possible -By future authors and reviewers

49. SASQAG 2/17/00 tomg@halcyon.com Generic Document Checklist PurposeHas the document satisfied the stated purpose? RedundantIs the same information presented elsewhere within the document? AcronymsHave all acronyms been defined prior to use? AccuracyIs the information wrong? ExtraneousIs there information outside the stated scope of the document? CompletenessIs necessary information missing? StandardDoes the document follow the xyz documentation standard?

50. SASQAG 2/17/00 tomg@halcyon.com Generic Document Worklist Before creating the deliverable... -Hold Kickoff Meeting Invite the right participants Plan for a successful document Build/Refine Checklists While working on the deliverable… -Use the checklists -Use Walkthroughs and Desk Checks as needed to reduce rework (major errors). When finished with the deliverable… -As per SQA plan, perform the appropriate Peer review -Send peer review data summary to SQA focal

51. SASQAG 2/17/00 tomg@halcyon.com For More Information

52. SASQAG 2/17/00 tomg@halcyon.com Resources... SASQAG WEB Site (sasqag.org) -Past Meetings -Links UW Certificate Course Commercial Courses SASQAG Training Courses

53. SASQAG 2/17/00 tomg@halcyon.com Goals and Objectives Some Terms Peer Reviews Kickoff Meetings Audits Checklist and Worklists Sharpen Your SQA Skills

54. SASQAG 2/17/00 tomg@halcyon.com