Coding concerns, are they real? Fadi Wedyan, Dalal Alrmuny May 10 th, 2007

Outlines Objective. Motivation. Study method. Results. Threats to validity. Conclusions. Future work and suggested projects. Questions.

Objective Investigating the possible connection between coding concerns and software failures in OSS.

Motivation No existing study attempted to answer the research question we asked. Many inspection tools exist, are they useful, are the concerns “real”?

Study method 1) choose the OSS to be studied, 2) choose tools for code inspection, and 3) develop a methodology to map the coding concerns to the reported failures

Research Questions Q1: “Does a relationship between coding concerns and reported failures in OSS exist?” Q2: “Are there certain types of coding concerns that are more likely to produce failures in OSS?”

Hypotheses First Hypothesis H1: Null hypothesis H01: “There is no relationship between coding concerns and reported failures in OSS”. Alternative hypothesis HA1: “A relationship exists between coding concerns and reported failures in OSS”. Second Hypothesis H2: Null hypothesis H02: “There is no relationship between certain types of coding concerns and reported failures in OSS”. Alternative hypothesis HA2: “A relationship exists between certain types of coding concerns and reported failures in OSS”.

Studied Systems Project NameReleasesNo. DevelopersKLOCNumber of Files jEdit5 stable, 6 mature iText4-beta, 5 stable Table 1: Main characteristics of the studied software systems

Code Inspection Tools ToolTypeAnalysisCode IntelliJ ® IDEACommercialStaticSource code FindBugs™OSSStaticBytecode Table 2: Main characteristics of code inspection tools used in the study

Sources of failure data Reported bugs in SourceForge –Incomplete. –Lot of faked bugs. CVS Repository –Complete. –Distinguish bugs fixing actions from others.

Results (1) VersionDate releasedReported failuresFindBugs™IntelliJ ® IDEA iText 1.3April 6, iText 1.4March 3, jEdit 3.0Dec 21, jEdit 4.1March 27, jEdit 4.2Aug 28, Table 3: Reported failures compared to concerns

Results (2) iText 1.3iText 1.4jEdit 3.0jEdit 4.1jEdit 4.2 Bad Practice Correctness Malicious Multithreaded45122 Dodgy Total Table 4: Coding concerns reported by FindBugs™

Results (3) Table 5: Coding reported by IntelliJ® IDEA iText 1.3iText 1.4jEdit 3.0jEdit 4.1jEdit 4.2 Class Structure Error Handling Numerical Issues Cloning Issues75444 Control Flow Issues Data Flow Issues Threading Issues Probable Bugs Total

Results (4) Table 6: Coding concerns categories compared to detected failures in iText 1.3 using IntelliJ® IDEA CategoryConcernsFailures Class Structure1030 Error Handling392 Numerical Issues141 Cloning Issues71 Control Flow Issues720 Data Flow Issues380 Threading Issues50 Probable Bugs6537 Total93111

Results (5) Table 7: Coding concerns categories compared to detected failures in iText 1.3 using FindBugs™ CategoryConcernsFailures Bad Practice1084 Correctness120 Malicious2340 Multithreaded40 Dodgy791 Total4465

Threats to Validity Internal validity: –The size of the studied systems. –The reported failures. External validity: –Size and type of the data sets. –Code inspection tools.

Conclusions A relationship exists between coding concerns and reported failures: the inspection tools were successful in pointing out 12-26% of the reported failures. Certain categories of concerns were successfully mapped into failures more than others.

Conclusions We recommend relying on the CVS repository instead of the reported failures in SourceForge. –About 40% of failures fixing found in CVS did not exist in the reported failures.

Future work and suggested projects More testing: extending the study on more software and using other tools. Studying early releases of software. Studying the “pattern of errors in Java codes”. –Results can be integrated in the inspection tools. For example, tracing the life cycle of objects.

Questions