1. DevCOP: A Software Certificate Management System for Eclipse Mark Sherriff and Laurie Williams North Carolina State University ISSRE ’06 November 10, 2006

2. 2 Agenda Background –Motivation and Hypothesis –Software Certificates and Software Certificate Management Systems (SCMS) DevCOP (Defect Estimation with V&V Certificates on Programming) –Research Goal and Methodology –DevCOP Certificates –The DevCOP SCMS Eclipse plug-in Limitations Current Status Questions

3. 3 Motivation Software Reliability –Often not estimated until development is complete –Actual reliability not known until system is shipped to customers Corrective action is more expensive later in the process If defect density could be estimated in-process… –Steps could be taken to address issues early –More economical, could improve development effort

4. 4 Hypothesis Defect density estimation can be based upon the history of verification and validation techniques that have been performed on the project. Questions that need to be answered: What is the best way to record V&V techniques? How do I build a model that can predict defect density with V&V information?

5. 5 Recording V&V Techniques Software Certificates –A record of a verification and validation (V&V) practice employed by developers and used to support traceability between code and evidence of the V&V technique used Software certificates could be any type of record: –Logs from test case runs –Reports from code inspections –Details on pair programming assignments However, these are all different forms of V&V information and maintaining this information could be expensive.

6. 6 Recording V&V Techniques Software Certificate Management Systems –Provides an interface and infrastructure to automatically create, maintain, and analyze software certificates –Benefits: Software maintenance Analysis of V&V technique effectiveness Reference in future projects Current research: –OGI/PSU: Programatica, a SCMS for Haskell

7. 7 DevCOP Research hypothesis: Defect density estimation can be based upon the history of V&V techniques that have been performed on the project. Methodology: Build a parametric model using software certificate information and previous project defect data to create a prediction model for future projects.

8. 8 DevCOP Certificates Records: –identifying information for the function it is associated with including its name, signature, class, and file location; –identifying information for the developer that created it; –the type of V&V technique used; –a hash of the function’s abstract syntax tree (AST); and –a significance weight.

9. 9 DevCOP Certificates Types of Certificates –Manual Includes all manual techniques, such as pair programming and code inspections –Automated Static Analysis Includes all techniques that can be run automatically on uncompiled code –Dynamic Includes all techiques performed automatically at run time, such as black box testing –Formal Includes all formal methods, such as lambda calculus and proofs

10. 10 DevCOP SCMS Eclipse Plug-in Currently supports manual V&V techniques and jcoverage certificates Provides different methods for examining and managing certificate data Demo

11. 11 Potential Side Effects Retrospective Causal Analysis –Once certificates are recorded on a project and bugs are reported, developers can use certificate and defect information to evaluate the efficacy of their V&V practices. Building certificate information in with compiled code base –If certificate information could travel with compiled code, it could be referenced at runtime so that other systems could evaluate whether it was to work with that system.

12. 12 Original Hypothesis Parametric Modeling: Method by which dependant variables are related to one or more independent variables with regard to previous data In Software Engineering… –Purpose is to provide an estimated answer to a software development question earlier in the development lifecycle Famous SE parametric models: COCOMO 81 and COCOMO II

13. 13 Original Hypothesis Early Research: Software Testing Reliability Early Warning –Uses a suite of metrics gathered on static code to provide a reliability estimate If a reliability estimate can be created from testing and static metrics, could it be improved if we added other verification and validation information to the model?

14. 14 Limitations of DevCOP Eclipse Environment –Even though numerous companies do use Eclipse, it might not be standardized Tool Selection –DevCOP uses JCoverage, which might not be the approved coverage tool Does add minimal overhead –Even adding one step can be too much…

15. 15 Evolution of Research The core theory behind DevCOP is that artifacts generated during software development can be useful during the development process. A certificate is simply an artifact of software development. Can we gather certificates or other artifacts with NO ADDITIONAL developer involvement?

16. 16 Evolution of Research SDAA: Software Development Artifact Analysis –By mining source control data, testing records, and other V&V information, new associations can be found between code units. –The type of association is determined by the type of development artifact used.

17. 17 Uses of SDAA Identification of Areas of Risk –Associations generated from defect data Impact Analysis –What files are likely to change together? Test Case Prioritization –How does a particular test relate to a given association?

18. 18 Thank you! Questions? Queries? Quandries? Contact Information: Mark Sherriff mark.sherriff@ncsu.edu DevCOP Project: http://agile.csc.ncsu.edu/devcop/