1. Sandeep Krishnan, ISU Robyn R. Lutz, ISU & JPL Katerina Goseva-Popstojanova, WVU Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 1 This research is supported by NSF grants 0916275 and 0916284

2. Background Product line – A family of products designed to take advantage of their common aspects and predicted variabilities [Weiss and Lai 1999] e.g. Nokia cellphones, HP printers, etc. Products - Commonalities – Shared by all products. e.g. Platform Variabilities – Differentiate the products High-reuse variation – JDT, PDE, Mylyn, Webtools, etc. Low-reuse variation – CDT, Datatools, Java EE tools. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 2

3. Motivation Does reliability improve as a product line evolves? Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 3 PL EvolutionReliability Reliability – Continuity of correct service; measured in terms of post-deployment failures [Avizienis, et al., 2001]

4. Research Questions Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 4

5. Findings Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? Common High-reuse Low-reuse Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 5

6. Approach Source of failure reports- Failure cateogries – Raw number of severe failures and percentage. Source of change reports – CVS repository of Eclipse. We look at change to existing files measured by KChanges and change via new files [Mockus et al., 2000]. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 6 BlockerCriticalMajorNormalMinor 20072008200920102011 EuropaGanymedeGalileoHelios BlockerCriticalMajor

7. Observations Number of severe failures decreases over time as expected Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 7 1. A. Failure trends – Common components

8. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 8 Percentage of severe failures tends to stabilize and even increases gradually 1. A. Failure trends – Common components

9. Observations Number of severe failures show mixed behavior, contrary to our expectations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 9 1. B. Failure trends – High-reuse variation components

10. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 10 Percentage of severe failures tends to stabilize and even increases gradually, contrary to our expectations 1. B. Failure trends – High-reuse variation components

11. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 11 Number of severe failures do not monotonically decrease; rather show mixed behavior 1. C. Failure trends – Low-reuse variation components

12. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 12 Percentage of severe failures show mixed behavior and not a decreasing trend 1. C. Failure trends – Low-reuse variation components

13. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 13 2. Change trends Common components: Percentage of new files decreases. High-reuse variation components: Percentage of new files tends to remain stable. Low-reuse variation components: Percentage of new files shows decreasing trend. All components show relatively high modification rate.

14. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 14 3. Failure/evolution relationship Failures/new-file o Common components: For two subcomponents, release 1 has lower failures/new-file than release 4. o High-reuse variation components: Release 1 always has higher failures/new- file than release 4. o Low-reuse variation components: No specific trend.

15. Observations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 15 3. Failure/evolution relationship Failures/Kchanges o No stable behavior for all components; mixed trends are observed.

16. Conclusion Common components have fewer severe failures over time and amount of change also decreases. Variation components show no uniform pattern of decrease in failures. Even high-reuse variation components continue to change fairly rapidly. Study suggests there may be no simple answer as to whether reliability tends to improve as product line matures. Further investigation is needed on the relationship between failure and change in software product lines. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 16

17. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 17