Sandeep Krishnan, ISU Robyn R. Lutz, ISU & JPL Katerina Goseva-Popstojanova, WVU Dept. of Computer Science, Iowa State University, MSR, May 22, This research is supported by NSF grants and

Background Product line – A family of products designed to take advantage of their common aspects and predicted variabilities [Weiss and Lai 1999] e.g. Nokia cellphones, HP printers, etc. Products - Commonalities – Shared by all products. e.g. Platform Variabilities – Differentiate the products High-reuse variation – JDT, PDE, Mylyn, Webtools, etc. Low-reuse variation – CDT, Datatools, Java EE tools. Dept. of Computer Science, Iowa State University, MSR, May 22,

Motivation Does reliability improve as a product line evolves? Dept. of Computer Science, Iowa State University, MSR, May 22, PL EvolutionReliability Reliability – Continuity of correct service; measured in terms of post-deployment failures [Avizienis, et al., 2001]

Research Questions Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Dept. of Computer Science, Iowa State University, MSR, May 22,

Findings Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? Common High-reuse Low-reuse Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse Dept. of Computer Science, Iowa State University, MSR, May 22,

Approach Source of failure reports- Failure cateogries – Raw number of severe failures and percentage. Source of change reports – CVS repository of Eclipse. We look at change to existing files measured by KChanges and change via new files [Mockus et al., 2000]. Dept. of Computer Science, Iowa State University, MSR, May 22, BlockerCriticalMajorNormalMinor EuropaGanymedeGalileoHelios BlockerCriticalMajor

Observations Number of severe failures decreases over time as expected Dept. of Computer Science, Iowa State University, MSR, May 22, A. Failure trends – Common components

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Percentage of severe failures tends to stabilize and even increases gradually 1. A. Failure trends – Common components

Observations Number of severe failures show mixed behavior, contrary to our expectations Dept. of Computer Science, Iowa State University, MSR, May 22, B. Failure trends – High-reuse variation components

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Percentage of severe failures tends to stabilize and even increases gradually, contrary to our expectations 1. B. Failure trends – High-reuse variation components

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Number of severe failures do not monotonically decrease; rather show mixed behavior 1. C. Failure trends – Low-reuse variation components

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Percentage of severe failures show mixed behavior and not a decreasing trend 1. C. Failure trends – Low-reuse variation components

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Change trends Common components: Percentage of new files decreases. High-reuse variation components: Percentage of new files tends to remain stable. Low-reuse variation components: Percentage of new files shows decreasing trend. All components show relatively high modification rate.

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Failure/evolution relationship Failures/new-file o Common components: For two subcomponents, release 1 has lower failures/new-file than release 4. o High-reuse variation components: Release 1 always has higher failures/new- file than release 4. o Low-reuse variation components: No specific trend.

Observations Dept. of Computer Science, Iowa State University, MSR, May 22, Failure/evolution relationship Failures/Kchanges o No stable behavior for all components; mixed trends are observed.

Conclusion Common components have fewer severe failures over time and amount of change also decreases. Variation components show no uniform pattern of decrease in failures. Even high-reuse variation components continue to change fairly rapidly. Study suggests there may be no simple answer as to whether reliability tends to improve as product line matures. Further investigation is needed on the relationship between failure and change in software product lines. Dept. of Computer Science, Iowa State University, MSR, May 22,

Dept. of Computer Science, Iowa State University, MSR, May 22,