Tom Gilchrist The Tools and Techniques of SQA SASQAG, February 17, 2000

SASQAG 2/17/00 Disclaimers The views and opinions expressed in this presentation are not necessarily those of my employer. Your results may vary. You are free to use any slide in your work, but if you do, please include my name and .

SASQAG 2/17/00 Goals and Objectives Some Terms Peer Reviews Kickoff Meetings Audits Checklist and Worklists Sharpen Your SQA Skills

SASQAG 2/17/00 Software Quality Assurance Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Checks/Audits) Review and measure the quality of software products and processes throughout development. (Peer Reviews) Provide software project management (and other appropriate parties) with the results of reviews and process checks. Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.

SASQAG 2/17/00 Validation and Verification Validation -Are we building the right product? Verification -Are we building the product right? Static Verification (of software) -Verification of a program by examination and analysis of its program elements.

SASQAG 2/17/00 Quality In Perspective

SASQAG 2/17/00 Customer vs. End User End User: -The person who uses the software product in the value stream Customer: -Anyone who is affected by the end product or intermediate software engineering deliverables being produced (reviewed).

SASQAG 2/17/00 Healthy Customer/Supplier Relationship Implies product use by customer! Implies choice Communications centers on expectations. -How we identify expectations -How we document and communicate them -How we measure/improve them

SASQAG 2/17/00 Value vs Quality Customers are concerned with value not just quality.

SASQAG 2/17/00 Value vs Quality Customers are concerned with value not just quality. "Value is not overpaying for quality." Value is..."the right combination of product quality, fair price, and good service." * From Business Week, Nov 11, 1991

SASQAG 2/17/00 Low Quality in Software Unneeded ComplexityUnneeded Complexity ReworkRework -Wrong -Unclear -Missing -Extra

SASQAG 2/17/00 The Project Requirements Finished Product SpecsDesignCodeTest Time Project Plan

SASQAG 2/17/00 Where is rework introduced? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

SASQAG 2/17/00 Where is rework found? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

SASQAG 2/17/00 Where is rework fixed? Requirements Finished Product SpecsDesignCodeTest Time Project Plan

SASQAG 2/17/00 The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981

SASQAG 2/17/00 The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981 High Visibility/Reward

SASQAG 2/17/00 The Quality Lever Requirements Design Coding Development Testing Acceptance Testing Improve Product Production X 30-70X 15-40X 10X 3-6X 1X Data from Boehm: Software Engineering Economics, 1981 High Visibility/Reward Low Visibility/Reward

SASQAG 2/17/00 Error/Defect Classification Number of Errors MinorMajor"Show Stoppers" Error Severity

SASQAG 2/17/00 Error/Defect Classification Number of Errors MinorMajor"Show Stoppers" Find and remove as many Major and Show Stopper errors as early as possible. Error Severity

SASQAG 2/17/00 Terminology Major ErrorMajor Error -Something that is wrong -High degree of probability that it will cause rework -Rework later will cost much more if left allowed to migrate downstream. Minor ErrorMinor Error An issue or defect that is not major. The costs of fixing later is not significantly higher than fixing now.

SASQAG 2/17/00 Quality Metric…Rework Number of Major Errors / Unit of WorkNumber of Major Errors / Unit of Work -ME/Page -ME/Drawing Zone -ME/KLOC -ME/?

SASQAG 2/17/00 Peer Reviews

SASQAG 2/17/00 Initial Level Informal Checking Do Work Down- stream Customers Input Source Materials Errors Found Process Product Entry Criteria Loosely Defined Exit Criteria For Quality Not Defined Work Process ENTRYENTRY EXITEXIT

SASQAG 2/17/00 Initial Level Informal Checking Do Work Down- stream Customers Re-Work Input Source Materials Errors Found Process Product Entry Criteria Loosely Defined Exit Criteria For Quality Not Defined Work Process ENTRYENTRY EXITEXIT

SASQAG 2/17/00 Measure and Improve Product Quality Do Work Down- stream Customers Re- Work Errors Found Process Product Crossfunctional Checklists SW Inspection Input Source Materials Entry Criteria Loosely Defined Exit Criteria For Quality Defined ENTRYENTRY EXITEXIT Work Process

SASQAG 2/17/00 Improve Product Quality and Reduce Variation Do Work Down- stream Customers Re- Work Errors Found Process Product Crossfunctional Checklists Input Source Materials Entry Criteria Defined Exit Criteria For Quality Defined ENTRYENTRY EXITEXIT Work Process SW Inspection

SASQAG 2/17/00 To Improve Product AND Process (Entry Criteria Met) Do Work Down- stream Customers Re- Work Input Source Materials Errors Found Process Product Crossfunctional Checklists Exit Criteria Met Statistical Data Analysis and Causal Analysis P D C A Deming Cycle (Defect Prevention Process) Repeatable Work Process SW Inspection

SASQAG 2/17/00 SW Inspection Process Flow Planning Collect and Store Data Another Chunk? Verify and Approve AuthorsModeratorInspection Team Members OK Re-Work YES NO Other Inspection and CI Processes Produce Product Rework Product Release Individual Preparation Overview Meeting Error Logging Meeting 7

SASQAG 2/17/00 SDLC ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation Time

SASQAG 2/17/00 SW Inspections ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation SW Inspection

SASQAG 2/17/00 Kickoff Meeting ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Program Element Creation Program Element Creation Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting SW Inspection

SASQAG 2/17/00 Task Kickoff Meeting Identify people/organizations affected by errors in the activity Identify and document potential major errors (build/modify checklists) Confirm task/process entry conditions Identify resources Forum for learning Reduce the amount of time needed by the author(s) to produce a deliverable by giving technical and resource information to do the job.

SASQAG 2/17/00 Kickoff Meeting ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Program Element Creation Program Element Creation SW Inspection

SASQAG 2/17/00 Walkthrough ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Walkthrough Program Element Creation Program Element Creation Walkthrough 21 Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting SW Inspection

SASQAG 2/17/00 Walkthrough Process Reduce the amount of time needed for re-work by reviewing work completed to date and address questions and concerns. Led by author Discussion of deliverable style and format Examine technical alternatives Evaluate checklists, standards, and rules. Capture undocumented issues Forum for learning

SASQAG 2/17/00 Peer Review Methods Walkthrough MethodsTypical GoalsTypical Attributes Minimal overhead Developer training Quick turnaround Little/no preparation Informal process Meetings No measurement Led by the author SW Inspection Detect and report all defects efficiently and effectively. Usage of deliverable Formal process Known Coverage Rate Moderator Checklists Customer Reviewers Measurement Desk Check Minimal overhead Quick turnaround Little/no preparation Informal process No measurement No Meetings Done by the author

SASQAG 2/17/00 SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting

SASQAG 2/17/00 SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check

SASQAG 2/17/00 SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check SW Inspection

SASQAG 2/17/00 SQA Interventions Output ENTRYENTRY EXITEXIT Input Program Element Creation StartDuringEnd Kickoff Meeting Walkthrough Desk Check SW Inspection Formal Walkthrough Formal Desk Check

SASQAG 2/17/00 Peer Reviews ENTRYENTRY EXITEXIT Output... ENTRYENTRY EXITEXIT Output Input Standards, templates, rules and checklists Standards, templates, rules and checklists Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Task Kickoff Meeting Walkthrough Desk Check Walkthrough Desk Check Program Element Creation Program Element Creation SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check SW Inspection Formal Walkthrough Formal Desk Check Walkthrough Desk Check Walkthrough Desk Check

SASQAG 2/17/00 Audits

SASQAG 2/17/00 Audit Process Software Quality Assurance is to provide management with appropriate visibility into the process being used by the software developers and of the products being built. Determine: Conformance to policies, procedures, and standards Adequacy of policies, procedures, and standards Effectiveness and efficiency of policies, procedures, and standards Assess personnel familiarity to requirements and documentation Assure availability, use and adherence to software standards

SASQAG 2/17/00 Audit Process Un- satisfactory Report? OK NO YES Produce Product Corrective Actions Audit Kickoff Meeting Develop Audit Checklist Conduct Audit DevelopersAuditor & Project Manager Write-up Report & Findings Auditor Identify Requirements Follow-up Audit Re-Work Close Audit & File END Start Review with Manager

SASQAG 2/17/00 Checklists & Worklists

SASQAG 2/17/00 Checklists Used to identify Major Errors Usually in the form of questions When used…. -By the authors to keep from inserting rework into the work stream -By reviewers to find and report as many Major Errors as possible -By future authors and reviewers

SASQAG 2/17/00 Generic Document Checklist PurposeHas the document satisfied the stated purpose? RedundantIs the same information presented elsewhere within the document? AcronymsHave all acronyms been defined prior to use? AccuracyIs the information wrong? ExtraneousIs there information outside the stated scope of the document? CompletenessIs necessary information missing? StandardDoes the document follow the xyz documentation standard?

SASQAG 2/17/00 Generic Document Worklist Before creating the deliverable... -Hold Kickoff Meeting Invite the right participants Plan for a successful document Build/Refine Checklists While working on the deliverable… -Use the checklists -Use Walkthroughs and Desk Checks as needed to reduce rework (major errors). When finished with the deliverable… -As per SQA plan, perform the appropriate Peer review -Send peer review data summary to SQA focal

SASQAG 2/17/00 For More Information

SASQAG 2/17/00 Resources... SASQAG WEB Site (sasqag.org) -Past Meetings -Links UW Certificate Course Commercial Courses SASQAG Training Courses

SASQAG 2/17/00 Goals and Objectives Some Terms Peer Reviews Kickoff Meetings Audits Checklist and Worklists Sharpen Your SQA Skills

SASQAG 2/17/00