1. The Value of Managing the Review Process
Mike Chapman

2. Intro Metrics Data Program NASA Glenn Research Center JPL
Michael Fagan

3. Definitions
Inspection – In software engineering this refers to peer review of any work product to look for defects.
Might also be called a Fagan Inspection
Common review practice
The goal is to identify and repair defects
Code Review – a kind of inspection that identifies defects in source code
Peer Review
Software walkthroughs
Software inspections

4. Elements of a Peer Review
Structured review process
Defined roles of participants
Forms and reports

5. The Standard Process 1. 2. 3. 4. 5. Review is scheduled End Review
Reviewers fill in
comments or defects 1. 2. 3.
Results are
summarized
Reviewers
convene,
discuss defects 4.
(Time can be tracked
throughout the process) 5.
End Review

6. The Process (cont) Instantiation Who What When Where Preparation
The key to success
Review
Resolution
Enter defects into tracking system
Assign action items

7. The Centrality of the Review Process
Affects every team member at some point
Impacts schedule
Impacts costs
Prevention of downstream costs
Cost of reviews
Impacts morale

8. Defect Removal Efficiency
Low Median High
Low Median High
No design inspections % % % Formal design inspections 95% 99% %
No code inspections Formal code inspections
No quality assurance Formal quality assurance
No formal testing Formal testing
Formal design inspections 43% 60% 68% Formal code inspections % % 66%
And nothing else and nothing else
Formal quality assurance % 45% 55% Formal testing % 53% 60%
Formal design inspections 70% 85% 90% Formal quality assurance % 65% 75%
Formal code inspections Formal testing
Formal design inspections % 97% 99% Formal code inspections % 87% 93%
Formal code inspections Formal quality assurance
Formal testing Formal testing
Software Quality by Capers Jones

9. Relative Cost to Fix Defects by Phase
1000
500
200
Larger software projects
100
Relative cost to fix defect 50 20 10
Smaller software projects 5 2 1
Requirements Design Code Development Acceptance Operation
test test
Phase in which defect was fixed

10. Software Defect Repair Rates
Non-test Defect Removal
Average 2.5
Origin
Lowest
Mode
Highest
Requirements Inspection
.25
2.0
16.0
Design Inspection
.50
6.0
24.0
Code Inspections
.10
Test Plan Inspection
.20
1.5
4.0
User Manual Inspection .6
3.0
Post-Release Defect Removal
Average 9.57
Origin
Lowest
Mode
Highest
Severity 1 Code Defect
8.00
18.0
40.0
Severity 2 Code Defect
5.50
10.5
Severity 3 Code Defect
.10
2.0
16.0
Severity 4 Code Defect
1.5
3.0
8.0
Abeyant
24.0
60.0

11. Savings from Performing Inspections
C error rates are very well documented: 13.7/KLOC (Open bracket to Close bracket)
100,000 line project – 1370 Errors
Defect removal Cost and Savings per 100,000 LOC by identification activity
Activity
Pre-Operations
Post Operations
Total Man Hours
$70/hour
Savings
Do nothing (40%)
548 * 3.5 = 1918
822 * 9.57 = 7866
9784/100 KLOC
$684,880
Testing (53%)
726 * 3.5 = 2541
644 * 9.57 = 6163
8704/100 KLOC
$609,280
$75,600
Testing and QA (65%)
912 * 3.5 = 3192
458 * 9.57 = 4383
7575/100 KLOC
$530,250
$154,630
Design or Code Reviews (60%)
822 * 2.5 = 2055
548 * 9.57 = 5244
7299/100 KLOC
$510,000
$174,880
Code Review and Test (70%)
959 * 3.0 = 2877
411 * 9.57 = 3933
6810/100 KLOC
$476,700
$208,180
Design Review and Test (80%)
1096 * 3.0 = 3288
274 * 9.57 = 2622
5910/100 KLOC
$413,700
$271,180
Code Review, Test and QA (87%)
1191 * 3.0 = 3575
179 * 957 = 1713
5288/100 KLOC
$370,162
$314,718
Design, Code Reviews & Test (97%)
1329 * 3.0 = 3987
41 * 9.57 = 392
4379/100 KLOC
$306,530
$378,350
Do this.
Save this!

12. So let’s run out and start doing inspections!
While the value of doing inspections has been proven, there is no consensus on a process.

13. Appropriate Processes
Reviews have a cost
Stay lean
Meeting size (Software lead, Group lead, developer, interfaces, QA,CM)
Meeting length (a maximum of 2 hours, major issues only)*
Schedule (stay absolutely stringent)
Automate the process
No one process is right for everyone
Be careful of dogma
Not-Invented-Here
Some groups need to own the process
* The number of defects identified was increased by 33% if there was a meeting.
 ``An Experiment to Assess the Cost-Benefit of Code Inspections in Large Scale Software Development", A.Porter, H.Siy, C.Toman,
and L.Votta, accepted for Third Symposium on the Foundations of Software Engineering, October 1995.

14. A Blog Excerpt JROLLER is a blog for the Java community.

15. Manifesto for Agile Software Development
Another Approach
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

16. The Automation of the Review Process
Saves time and reduces frustration for the team
Helps limit costs
Archives results for future retrieval
Really its essential if you are going to take advantage
of the quantitative output

17. Automates the review process Makes the review process more efficient
Provides process metrics on the review cycle
Takes this part of your development to CMMI Level 4
Remove the pain from your review process!

18. Integrated Software Metrics, Inc.
Contact Info
Integrated Software Metrics, Inc.
Mike Chapman