Formal Technical Reviews Matt Graham 18 November 2004 EECS 814 University of Kansas

Outline  Introduction  Generic Inspection and Variations  Inspection throughout Development Cycle  FTR Metrics  Tools  Conclusion

FTR: What it is...  A method involving a structured encounter in which a group of technical personnel analyzes an artifact according to a well-defined process. ::Output::  A structured artifact that assesses or improves the quality of the artifact as well as the quality of the method. ::Input::

FTR: What it is... A structured FTR is a forum for:  Finding defect information for the Author.  Educating peers about product.  Providing fault likelihood data for testers.  Providing detailed status report for managers.  Allowing process improvement group a test to measure.

FTR: What it isn’t...  Walk-through  No Measurements  Often informal, impromptu  Main purpose is developer training

Why Review?  Improves overall product quality  Early detection of defects  Reduces rework and its associated costs  Educate the participants and provide training  Setting standards of excellence / maintain process improvement momentum  Improve schedule performance  IBM reported 83% and AT&T 92% defect detection through inspections

Inspection Process Overview Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  6 Steps  Team  members  Experienced and involved in product development  Distinct and important roles

Inspection Process Overview Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Roles:  Organizer – plans the inspection activities  Moderator – ensures that procedures are followed and moderates the meetings.  Inspectors – responsible for detecting defects in the product  Presenter – presents the product in logical fashion paraphrased at a suitable rate  Author – author of the developed product.  Recorder – records the defects during the meeting  Collector – collects the defects if there is no meeting.

Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – organize inspection  Check to see if work products pass the entry criteria  Select inspection participants and assign roles  Schedule inspection meeting  Distribute inspection material  Prepare inspection documentation  Ensure product inspection readiness

Objective Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – educate participants on the product being inspected  Author explains the inspection materials  Can be beneficial when:  The inspection artifact is complex  If the artifact is part of large system  If new members participate in inspection  Could be rolled into previous or next phase if product is simple enough  Communicate inspection goals!

Defect Detection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – to find defects  Can be divided into 2 phases:  Preparation (individual) phase.  Meeting (group) phases.  Individual preparation with the goal of detecting defects can help inspectors to be more prepared.

Defect Collection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – collectively agree and document the defects (triage)  Decide on further inspection needed or not  Decisions made subjectively as a group.

Defect Collection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – correct the defects collected in the previous phases  The author makes the detected changes and sends it for follow-up

Follow Up Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – to make sure all defects are resolved  Only one person involved and he/she verifies the defect resolution  Verifier should “return and report” to inspection team

Formal Inspection Methods  Fagan Inspection  Asynchronous Inspection  Phased Inspections  N-Fold Inspections

Fagan Inspection Overview Preparation Inspection Rework Follow-Up  4 Roles  Author  Reader  Moderator  Scribe  Meeting centric – cost of scheduling and time  Meetings add little to defect detection

Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  Team  Author not involved  Moderator and Inspectors Re-Work

Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  No meetings  Easy to assess participation  Process improvement from documenting all correspondences  Allows parallel communication  Can be distributed in space and time  Eliminates group approval Re-Work

Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  Moderator sends out material  Initial Individual Review – create list of defects  Circulate the copy of defect list to all inspectors and discuss via  Individual Review – update defect list and send to Moderator  Moderator compiles final defect list, send it to author and follow up – eliminates group approval Re-Work

Phased Inspections  Consists of several coordinated partial inspections called phases  Each phase inspects for a specific property or small set of related properties  Each phase responsible for thorough checking of the properties  Some phases have single inspectors and others have multiple inspectors  Reference documents are provided to inspectors at the beginning of each phase and each inspects using checklists  Domain specific checklists  Application specific checklists

N-fold Inspection  N independent teams inspect the same product using traditional inspection method  Collective effort of multiple teams more faults than a single team  Moderator collects faults from independent teams and composes the final defect list

Requirements Inspection  Reading Techniques are techniques to analyze work product during defect detection  Ad-hoc  Checklist based reading  Scenario based readings  Perspective based reading  Defect based reading

Perspective based Reading  Inspectors stand in for specific stakeholders in the document to verify the quality of requirements.  Designer – detail for creating system components  Tester – detail for constructing test plans  User – correct functionality  For each perspective, reviewer creates high level work products from the requirements document

Code Inspections  Cognitive models for program comprehension  Bottom-up  Top-Down  Integrated  Systematic and as-needed comprehension

FTR Metrics  Helps you measure the effectiveness of the inspection  Aids in continuous process improvement  Provides feedback to management  A unit of work product varies often:  Usually 1000 lines of source code (not including comments)  Page counts and line counts for text

Some Generic Metrics  Average preparation effort and preparation rate per unit of material  Average examination effort per unit of material  Average explanation rate per unit of material  Average number of defects and major defects found per unit of material  Average hours per defect and per major defect

Nine Key Metrics by AT&T for Code Inspections  Total Non Comment lines of source code inspected, in KLOC  Average lines of code inspected  Average preparation rate  Average inspection rate  Average effort per KLOC  Average effort per fault detected.  Average faults detected per KLOC  Percentage of re-inspections  Defect-removal efficiency

Tools  Various tools for different inspection methods.  ICICLE – for inspection of C & C++ programs  Scrutiny & InspeQ for specific inspection processes  ASSIST –supports generic inspection process

Choosing a tool...  Threads of discussion  Sharing of Information  Train of thought  Visual Cues  Reaching a consensus  Coordination

The Future of FTR  FTR adoption is slow:  Managers see added cost of inspections instead of the benefits of greatly reduced defect leakage  Practitioners already have tight schedules and reluctant to take on additional responsibilities  Companies postpone adoption since peer reviews are part of CMM level 3.

FTR Barriers  Insufficient preparation  Moderator domination  Incorrect review rate  Ego-involvement and personality conflict  Issue resolution and meeting digression  Recording difficulties and clerical overhead

Conclusion  Organization’s size, culture and industry should be considered in deciding on the FTR method to use.  Review does not replace testing but can make it easier  Success of inspection lies in:  Process Adherence  Systematic Reading technique  Optimization

FTR: References ● Philip Johnson: ● Tom Gilb: ● IEEE STD