The Value of Managing the Review Process Mike Chapman

Intro Metrics Data Program NASA Glenn Research Center JPL Michael Fagan

Definitions Inspection – In software engineering this refers to peer review of any work product to look for defects. Might also be called a Fagan Inspection Common review practice The goal is to identify and repair defects Code Review – a kind of inspection that identifies defects in source code Peer Review Software walkthroughs Software inspections

Elements of a Peer Review Structured review process Defined roles of participants Forms and reports

The Standard Process 1. 2. 3. 4. 5. Review is scheduled End Review Reviewers fill in comments or defects 1. 2. 3. Results are summarized Reviewers convene, discuss defects 4. (Time can be tracked throughout the process) 5. End Review

The Process (cont) Instantiation Who What When Where Preparation The key to success Review Resolution Enter defects into tracking system Assign action items

The Centrality of the Review Process Affects every team member at some point Impacts schedule Impacts costs Prevention of downstream costs Cost of reviews Impacts morale

Defect Removal Efficiency Low Median High Low Median High No design inspections 30% 40% 50% Formal design inspections 95% 99% 99.99% No code inspections Formal code inspections No quality assurance Formal quality assurance No formal testing Formal testing Formal design inspections 43% 60% 68% Formal code inspections 43% 57% 66% And nothing else and nothing else Formal quality assurance 32% 45% 55% Formal testing 37% 53% 60% Formal design inspections 70% 85% 90% Formal quality assurance 50% 65% 75% Formal code inspections Formal testing Formal design inspections 85% 97% 99% Formal code inspections 75% 87% 93% Formal code inspections Formal quality assurance Formal testing Formal testing Software Quality by Capers Jones

Relative Cost to Fix Defects by Phase 1000 500 200 Larger software projects 100 Relative cost to fix defect 50 20 10 Smaller software projects 5 2 1 Requirements Design Code Development Acceptance Operation test test Phase in which defect was fixed

Software Defect Repair Rates Non-test Defect Removal Average 2.5 Origin Lowest Mode Highest Requirements Inspection .25 2.0 16.0 Design Inspection .50 6.0 24.0 Code Inspections .10 Test Plan Inspection .20 1.5 4.0 User Manual Inspection .6 3.0 Post-Release Defect Removal Average 9.57 Origin Lowest Mode Highest Severity 1 Code Defect 8.00 18.0 40.0 Severity 2 Code Defect 5.50 10.5 Severity 3 Code Defect .10 2.0 16.0 Severity 4 Code Defect 1.5 3.0 8.0 Abeyant 24.0 60.0

Savings from Performing Inspections C error rates are very well documented: 13.7/KLOC (Open bracket to Close bracket) 100,000 line project – 1370 Errors Defect removal Cost and Savings per 100,000 LOC by identification activity Activity Pre-Operations Post Operations Total Man Hours Cost @ $70/hour Savings Do nothing (40%) 548 * 3.5 = 1918 822 * 9.57 = 7866 9784/100 KLOC $684,880 Testing (53%) 726 * 3.5 = 2541 644 * 9.57 = 6163 8704/100 KLOC $609,280 $75,600 Testing and QA (65%) 912 * 3.5 = 3192 458 * 9.57 = 4383 7575/100 KLOC $530,250 $154,630 Design or Code Reviews (60%) 822 * 2.5 = 2055 548 * 9.57 = 5244 7299/100 KLOC $510,000 $174,880 Code Review and Test (70%) 959 * 3.0 = 2877 411 * 9.57 = 3933 6810/100 KLOC $476,700 $208,180 Design Review and Test (80%) 1096 * 3.0 = 3288 274 * 9.57 = 2622 5910/100 KLOC $413,700 $271,180 Code Review, Test and QA (87%) 1191 * 3.0 = 3575 179 * 957 = 1713 5288/100 KLOC $370,162 $314,718 Design, Code Reviews & Test (97%) 1329 * 3.0 = 3987 41 * 9.57 = 392 4379/100 KLOC $306,530 $378,350 Do this. Save this!

So let’s run out and start doing inspections! While the value of doing inspections has been proven, there is no consensus on a process.

Appropriate Processes Reviews have a cost Stay lean Meeting size (Software lead, Group lead, developer, interfaces, QA,CM) Meeting length (a maximum of 2 hours, major issues only)* Schedule (stay absolutely stringent) Automate the process No one process is right for everyone Be careful of dogma Not-Invented-Here Some groups need to own the process * The number of defects identified was increased by 33% if there was a meeting.  ``An Experiment to Assess the Cost-Benefit of Code Inspections in Large Scale Software Development", A.Porter, H.Siy, C.Toman, and L.Votta, accepted for Third Symposium on the Foundations of Software Engineering, October 1995.

A Blog Excerpt JROLLER is a blog for the Java community. http://jroller.com/page/kfurb?entry=committed_to_inspections

Manifesto for Agile Software Development Another Approach Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

The Automation of the Review Process Saves time and reduces frustration for the team Helps limit costs Archives results for future retrieval Really its essential if you are going to take advantage of the quantitative output

Automates the review process Makes the review process more efficient Provides process metrics on the review cycle Takes this part of your development to CMMI Level 4 Remove the pain from your review process!

Integrated Software Metrics, Inc. Contact Info Integrated Software Metrics, Inc. www.ismwv.com Mike Chapman mchapman@ismwv.com 304-657-7187