Copyright © 2005 QA Insight, Inc. All rights reserved. 1 A Review of Software Inspection Techniques Getting Higher Returns from Your Review Processes Karina Gamble, QA Insight, Inc Presented For SCQAA Inland Empire Chapter January 12, 2006

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Agenda  Introduction  Review Types – similarities and differences  Benefits of software inspections/reviews  Reasons for not establishing company-wide review processes  Recommendations and solutions

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Introductions  Founder of QA Insight, Inc.  Specialties: establishing processes, testing, training and mentoring – soon to launch eMentoring  Co-founder of the San Fernando Valley chapter of SCQAA serving the northern LA county  Our SCQAA overall objective  to provide educational talks educating people in various roles or job functions – Testers, PMs, BAs, and yes even developers towards achieving a common goal  Improve Software Product Quality

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis  Static vs. dynamic  Static means visual examination not examination by execution  Static analysis is sometimes improperly termed as static testing based on:  IEEE std. Glossary  Static analysis does not have to be 100% manual, in fact you need to use tools to help (not replace) your analysis process

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis Types Inspections Peer Reviews Passarounds /Deskchecks Walkthroughs Ad Hoc Reviews Pair* Programming Most Formal Least Formal Can be considered as an informal review type but more of a s/w development style (by Karl E. Weigers)

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Fagan’s Inspection Process Moderator  Planning  Overview  Preparation  Meeting  Rework  Follow-up Inspectors  Overview  Preparation  Meeting  Follow-up Author  Overview  Meeting  Rework  Follow-up Reader  Overview  Meeting  Follow-up Recorder  Scribe  Defect entry

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Inspections  Most formal – same as formal inspection  Developed by Fagan - IBM in 1976  Multiple roles in the review team:  Moderator, Inspector(s), Author, Reader, Recorder  Several sequential activities  Author is not the moderator and does not lead the meeting  Reader goes over small chunks of work product at a time in the meeting

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Peer Reviews  Less formal but still planned  Participants still get materials to review before the meeting  No overview or follow-up meetings  Author may lead the meeting (unlike inspections)  Not as efficient in finding defects as inspections are  Bigger chunks are reviewed

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Walkthroughs  Informal review meeting led by the author  Primarily used for education and soliciting feedback  Error detection happens during the meeting not during preparation phase  Participants are not expected to be familiar with code or design, or any other item under review  Level of detail reviewed is up to author’s discretion

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Passarounds  Informal review  Good way to start a review culture  Find people in your team that you respect and trust to review  Distributing product to review to multiple people at the same time  No meeting is held – just independent review  Each reviewer gets to see the comments of others to minimize redundancy  Can end up with very hot debates – Be careful!

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Yet Another Famous Cost of Defects Slide!

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Benefits of Inspections over Testing  Inspections are better in finding defects than just testing  Symptoms of problems instead of problems  Testing alone cannot tell you how maintainable or clear the code is

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Benefits of Code Inspections  Numerous studies have shown the benefits of static analysis  HP’s inspection program measured an ROI of 10 to 1  Inspection reduced the cost of finding error by factor of 10 at AT & T Bell labs.  Studies have shown the benefits of code inspections  65% errors found from Fagan’s inspections  35% errors found from tests  3.25 errors/unit effort of inspection  0.44 errors/unit effort of testing  Fagan inspections are 7.4 times more productive than testing! ( Note: You can use defects founds in inspections to predict defects remaining)

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Benefits of Inspections for Testers  “Testers can spend more of their time finding more subtle bugs Instead of finding bugs that developers should have found or better yet, should have prevented from introducing into code/design to begin with.”

Copyright © 2005 QA Insight, Inc. All rights reserved. 15 So Why Doesn’t Every software group do Inspections? Who Does Software Inspections???

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Reasons  General lack of knowledge about reviews  Not enough training  Cultural inhibitors (attitudes and past experiences)  Improper planning  Improper use of review metrics

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Why are Code Inspections often not being done?  Very time consuming  Code is too complex  Reviewers are usually not prepared  Manual inspection of OO programs is not easy  It is a manual process that you need to rely on the expertise and experience of your reviewers  Inconsistent results  Limited on how much code you can inspect

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Our Challenges  QA teams tend to focus more on testing - not by choice sometimes  Difficult to break territorial and cultural barriers  Management does not see the benefits, so no support for reviews and inspections  Lack of knowledge and tools  Myths and misconceptions – Management by opinions rather than by facts – no metrics  Status-quo vs. real positive changes

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. So What do we do now?  Training! Training! Training!  QA managers justify reviews to your organizations  QA managers define and document inspection/review processes in the QA plan  Collect review metrics  Summarize defect data and report  QA along with the DEV team to communicate the benefits  Do root cause analysis

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Tools and Techniques  Static analysis tools and reading techniques  Static Analysis tools:  To save time during code inspections and increase productivity run a code checker tool first – check for violations against standards, memory leaks, unhandled exception  These tools still don’t replace manual inspections  Reading Techniques are very useful for reviewing requirements  Helps reviewers find more defects efficiently

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis Tools  Automate some of the manual checking  Coding standards  Memory leaks  Uncaught runtime exceptions  Race conditions – different threads accessing the same variable  Deadlock conditions  Security vulnerabilities  Example tools: Jtest, Jlint, PMD

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis with JTest by Parasoft  Checks for 380 coding standards and automatically corrects the rule violations  Checks for a specific comment structure format validating the comment matches with the code – JContract  Automates unit testing – Black box testing at the unit (class) level

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis with Jlint  Jlint will check your Java code and find bugs, inconsistencies and synchronization problems by doing data flow analysis and building the lock graph.  Finds unreachable code  Threading/lock problems  More than just coding standard checking  Find bugs that even manual inspections can’t find – not even by experienced staff!  Jlint is extremely fast

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Static Analysis with PMD  PMD scans Java source code and looks for potential problems like:  Empty try/catch/finally/switch blocks  Unused local variables, parameters and private methods  Empty if/while statements  Overcomplicated expressions - unnecessary if statements, for loops that could be while loops  Classes with high Cyclomatic Complexity measurements

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Reading Techniques  Formal software inspections (e.g. Fagan’s)  Focus on structure, frequency of meetings  Organizational aspects  Not on technical aspects – how to review  Use Ad hoc reading techniques  Reading techniques  Optimize inspections – find most defects with less efforts  Tackle the technical aspects  Provide a structured process to help the groups improve their review process  Can help you find out what defects passed thru the process and what to do to improve the process

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Reading Techniques  Ad Hoc  No structure in place  Everyone attempts to look for all classes of defects  Checklist-based  Set of questions under each review category  One checklist by all inspectors  Scenario-based /Perspective-based – each reviewer gets to review artifact based on his/her own role or based on specific set of usage scenarios

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Scenario/Perspective Based  Each reviewer assumes a specific perspective – e.g. tester, designer and customer, maintainer, etc.  Reviewers are also required to produce a high level work products – not passive reading  Specific goals and questions for each perspective or scenario

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Reading Techniques Benefits  Systematic  Focused  Goal-oriented  Transferable via training  Inconclusive studies, however, as to whether or not PBR is significantly better than CBR

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Conclusion  Inspections are better and cheaper in finding defects than testing alone  Earlier detection of defects are possible by inspections  Manual inspections do take a lot of time and may not catch all defects for complex multi-threaded OO software  Static Analysis tools and Reading Techniques alleviate some of these problems  QA plays a key role in leading the inspection process and educating staff in processes, procedures, static analysis tools and in reading techniques

10/1/ Copyright © 2005 QA Insight, Inc. All rights reserved. Links    “How Perspective-Based Reading Can Improve Requirements Inspections” - IEEE Software July 2000  Open source tools:  IEEE Software Engineering BOK:  “Peer Reviews in Software, A Practical Guide” – Karl E. Weigers  Automated Requirement Measurement Tool available for free – see me or send me an for more details  (Karina is programs and education chair – we meet every 1 st Wednesday of the month)