What Type of Defects are Really Discovered in Code Reviews. Mika V What Type of Defects are Really Discovered in Code Reviews? Mika V. Mantyla and Casper Lassenius IEEE Transactions on Software Engineering Vol 35, issue 3, pp 430-448, May-june 2009. Presented by Meenakshi Lakshmikanthan on 03/17/2010

Main Idea Code Reviews are an integral part of the software engineering, but the knowledge of their benefits is incomplete. Previous research provide only a limited view of their effectiveness. This paper attempts to qualitatively explore and classify defects discovered in code reviews.

Literature Review Many peer review literature have focused on defect counts and overlooked qualitative issues and only a very few of them have looked deeper into the type of defects. Therefore, the literature has created the impression that code reviews are mainly about finding functional defects thereby giving an incomplete picture of the benefits of code reviews. The authors try to get a deeper understanding of the type defects found during the code reviews.

Literature Review Chillarege created a defect classification consisting of eight defect types and a distribution of 300 defects. Fagan’s work provided a schema of defect classification and distribution but the details of the classification are not clear. Russell classified the defects as major, minor and commentary making the classification quite coarse. Basili and Selby provided well-defined and often cited 2D classification. Unfortunately, the number of defects in the study was only 34, limiting its usefulness as a source for defect distributions.

Literature Review Siy and Votta proposed that 75% of the defects found in code reviews are evolvability defects that affect future development efforts instead of runtime behavior. But their research had some limitations like lack of detailed analysis, lack of descriptive information etc. The authors found it very odd that very few of the studies included defect types for evolvability defects; in other words either the evolvability defects have been classified under another defect class or such defects did not exist in the code.

Research Questions What is the distribution between functional defects and evolvability defect? Study the proposition by Siy and Votta that most defects discovered in code reviews are evolvability defects rather than functional defects. What different types of evolvability and functional defects are found in code reviews and how can they be classified based on defects’ technical content?

Type of Defects Functional Defect  A defect in the code that may cause system failure when the code is executed. Evolvability Defect  A defect in the code that makes the code less compliant with standards; more error prone and more difficult to modify extend or understand.

Data Collection Industrial Code Reviews The authors studies the code reviews of a company which was selected based on its accessibility and the fact that the company had established code review practices. Before each review, the responsible developer performed smoke testing. Prior to the review meeting, the reviewers read the code and tried to find quality deviations. The company had code review checklist but did not use them actively. The reviewers were mostly from the same development team.

Data Collection Student Code Reviews For this study, the authors had students in software quality assurance course from their university perform code reviews . 87 students participated in the code reviews with different levels of programming skills and code review experience. The student performed the reviews individually and submitted their individual defect logs. A checklist was available to the students but was not made compulsory to use the checklist.

Findings In both the industrial and student code reviews, over 70 percent of the findings were evolvability defects. In the industrial reviews, about 20 percent of the identified defects were functional, while in the student reviews, only 13 percent were functional. The ratio of evolvability defects to functional defects ranged between 5:1 and 3:1.

Distribution of Defects

Data Analysis and Classification

Classification The classification is based on literature survey and empirical data. The classification of a defect as evolvability or functional was based on whether the defect had an impact on the functionality. False positives were issues that were identified in the meeting but that were discovered not to be defects either during the meeting or after.

Evolvability Defects Evolvability defects have been categorized as Documentation : Commenting and naming of software elements, functions and classes. Textual Supported by Language Visual Representation : Defects hindering program readability for the human eye. Structure : Indicates the source code composition parsed by the compiled into a syntax tree. Organization : Defects that can be fixed by applying structural modifications to the software. Solution : These defects require an alternative implementation method.

Functional Defects To address the shortcomings of the existing functional defect classification, the authors created a functional defect taxonomy based on the research data and existing classifications. This classification of functional defects is seen as a superset of the previous classifications rather than a completely new one.

Functional Defects Resource Defects : Mistakes made with data or other resource initialization. Check Defects : Validation mistakes or mistakes made when detecting an invalid value. Interface Defects : Mistakes made when interacting with other parts of the software. Logic Defects : Defects made with comparison operations, control flow, and computations and other types of logical mistakes. Timing Defects : Defects that are possible only in multithread applications where concurrently executing threads or processes use shared resources. Support Defects : Relate to support systems and libraries or their configurations. Larger Defects: Defects in which functionality is missing or implemented incorrectly.

Distribution between Functional Evolvability Defects What is the distribution between functional defects and evolvability defects? Based on this study, roughly 75 percent of the findings identified in the code reviews were evolvability defects that would not cause runtime failures. What different types of evolvability and functional defects are found in code reviews, and how can they be classified based on the defects’ technical content? For the functional defects, the authors were able to confirm that the existing classifications match well. For evolvability defects, the authors have created a new classification.

Functional Defects Vs Evolvability Defects

Implications Code reviews seem valuable for software product or service businesses in which the same software or service is modified and extended over time. Only 10 % of the evolvability defects belong to the visual representation group that can be automatically detected and fixed. An established and empirically grounded defect classification should be used as a baseline when building checklists for code reviews or when creating coding standards for the developers.

Implications to Scenario-Based Reviews Documentation Reviewer : Approach the code from a textual point of view and try to determine whether it contains enough information in names, comments, and code element specification. Code Structure Reviewer : They should focus on structural organization defects. Solution approach Reviewer : Should think of alternative implementation approaches. Resource and Interface Reviewer : concentrate on functional defects belonging to the resource and interface categories. Check and Logic Reviewer : Focus on functional defects belonging to the check and logic categories.

Conclusion This paper affirms that code reviews are a good tool for detecting code evolvability defects that cannot be found in later phases of testing because they do not affect the software’s visible functionality. Second, this paper has provided the most comprehensive qualitative classification of code review findings to date.

Pro’s and Con’s The authors have done a great job of linking defect severity to software attributes . Based on their analysis, the authors have suggested ways to put their research findings into practice in real time industries. These findings will be of great use to product based companies whose products need to withstand evolution. This classification will be very useful while creating company coding standards and code review checklists. Also, the classification can provide input for creating automated defect detectors or developing new programming languages. Also, this paper has showcased the importance of finding evolvability defects and fixing them even though the customer can never see those directly.

Pro’s and Con’s The analysis has been based on the code review practices of just one company and hence the credibility is questionable. Though they have the student reviews as a secondary source of analysis, these reviews cannot be entirely relied as the pressure of grades might have affected the students as they may have tried to get as many defects as possible which might have resulted in more evolvability defects. The impact of individual evolvability defects are not known because it is possible that many of the evolvability defects mentioned could turn to be false positives during the long run. On the flip side, if the number of functional defects found during code review is less companies undertaking short term projects may have to think twice about investing time and money into the code review process.

Thank You!!!!!!!