1. Studying the test process in open source and industrial software systems Andy Zaidman, Bart Van Rompaey, Serge Demeyer, Arie van Deursen TU Delft, University of Antwerp Andy Zaidman, IPA Herfstdagen 24 th November 2008

2. How does test code * co-evolve? WHY? HOW? (*) Developer tests

3. Well, why would you… … want to know the “quality” of the tests? When maintaining the system…  can you trust the existing tests/test process? When judging the overall quality of a software system…  is it good enough for our standards? When monitoring the test process…  Comparing intended with actual process

4. You might say … “What about test coverage?” Nice, but somewhat hollow measure Easy to keep artificially high What about boundary testing? “Exercising versus testing…” Test process?

5. How?

6. Repository mining Software developers like to use VCS – e.g., CVS, SVN, Clearcase, MS Visual Sourcesafe, … – Backing up their work – Making it easy to work in groups – Easy to manage releases On the other hand… – A wealth of information is stored in there – “It’s like a free museum” of past development activities and development decisions – Let’s use this!

7. “Test health” Determine the long-term quality of your test suite. Don’t only look at the “testing product”, but also at the “testing process” – Tells something about the future of the testing (e.g., maintenance of the tests) “Test health” was introduced by Kent Beck

8. “Test health” VCS views Change History View – When are tests adapted? Growth History View – Determine impact of changes to production and test code. Test Writing Efficiency View – How much test code for coverage?

9. 3 subject software systems Checkstyle – Java source-code style checker (and improver) – ~ 6 years development, 2260 commits, 6 developers, 738 classes, 47 kSLOC ArgoUML – UML drawing application – ~ 7 years development, 7477 commits, 42 developers, 1533 classes, 130 kSLOC 1 industrial system – Software Improvement Group (SIG), Amsterdam, The Netherlands

10. Example change history view

11. Checkstyle Change History

12. SIG Change History “CS”

13. Checkstyle Growth History %

14. ArgoUML growth history view

15. SIG Growth History

16. Growth History co-evolution scenarios

17. Checkstyle test writing efficiency

18. SIG test writing efficiency

19. Test writing efficiency patterns

20. Conclusion? In our case studies: – Big difference between OS systems and SIG! – SIG Continuous testing effort eXtreme Programming + test-driven development – OS Sometimes 6 months of no testing No increase in testing before major release – But sometimes after! Buy 2.1 instead of 2.0!!! ArgoUML project was started without testing process in place

21. Next steps in this research Try this on more (diverse) software systems – e.g., OS projects from Apache group Try to correlate the testing effort to the number of bugs reported in Bugzilla – More bugs when testing effort is less good, and vice versa? More information? – See our ICST 2008 paper (it’s included in your IPA Fall Days welcome pack!) – An STVR journal paper is currently in submission

22. ? a.e.zaidman@tudelft.nl http://www.st.ewi.tudelft.nl/~zaidman a.e.zaidman@tudelft.nl http://www.st.ewi.tudelft.nl/~zaidman