1. CONTINUOUS INTEGRATION AND TEST Stephen Oglesby - CSCI577b – April 2011

2. Introduction  Who am I?  What is Continuous Integration?  Continuous Integration + Automated Testing  Requirements for successful CI&T  Scaling options  CI&T within CSCI577ab “Cake, and grief counseling, will be available at the end of the testing period.” - GLaDOS 4/27/2011 2

3. Continuous Integration  “Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” – Martin Fowler  Born out of eXtreme Programming (XP), not limited to it. 4/27/2011 3

4. What does that mean?  Code commit detection  Per check-in compilation (yes you should do this anyway, but now it’s automatic)  Automated build solution  Expandable to include static code analysis  Results reported to the team  Immediate notification of integration errors  Faster resolution 4/27/2011 4

5. CI & Test  Automated unit & acceptance testing is a natural extension of CI  Reduces test time and cost  Improve communication & project visibility  Updated and available test results throughout development  Bug Tracking System integration for automatically logging new defects  Reduce risk 4/27/2011 5

6. Requirements  Single source repository  Automated build  Automated self-test  Tests run on production environment clone 4/27/2011 6

7. Requirements (cont.)  Fast builds 4/27/2011 7

8. Scaling Up from XP  Add Hierarchy  Execute build at each level of integration  Label version units based on integration test depth Copper Silver Gold  Add Builds  Stable, Beta, Dev, Nightlies 4/27/2011 8

9. CI&T in CSCI577ab  Requires bringing test activities into the first semester  Test case definition during requirements gathering/WikiWinWin  Still requires test case dependencies, even more critical since automation environment needs them  Standardized test procedures and results, automatically generated  Automation Framework + CI Server should be largely reusable across most projects 4/27/2011 9

10. CI&T in CSCI577ab (cont.)  Provides quantitative assessment criteria  Percentage of development time with a failed build  Number of overnight build failures  Number of deleted test cases  Percentage of requirements with corresponding test cases  Percentage of methods not tested  Increases team collaboration, code quality, developer productivity 4/27/2011 10

11. Open Source CI&T Solutions  Automated Test Frameworks  Framework for Integrated Test (Fit + Fitnesse)  Software Testing Automation Framework (STAF)  Selenium  Continuous Integration Server  CruiseControl + Subversion “At the end of the experiment you will be baked, and then there will be cake” - GLaDOS 4/27/2011 11

12. References  Sean Stolberg, “Enabling Agile Testing Through Continuous Integration”, Agile Conference 2009, IEEE, pg 369-374, 2009  Hehui Liu, Zhongjie Li, Jun Zhu, Huafang Tan, and Heyuan Huang, “A Unified Test Framework for Continuous Integration Testing of SOA Solutions”, 2009 International Conference on Web Services, IEEE, pg 880-887, 2009  Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Test Automation Framework for Implementing Continuous Integration”, Sixth International Conference on Information Technology: New Generations, IEEE, pg 784-789, 2009  Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Implementing an Effective Test Automation Framework”, 23 rd Annual IEEE International Computer Software and Applications Conference, IEEE, pg 534-538, 2009  Jon Bowyer and Janet Hughes, “Assessing Undergraduate Experience of Continuous Integration and Test- Driven Development”, 28 th International Conference on Software Engineering, ACM, pg 691-694, 2006  Fabrizio Cannizzo, Robbie Clutton, and Raghav Ramesh, “Pushing the Boundaries of Testing and Continuous Integration”, Agile 2008 Conference, IEEE, pg 501-505, 2008  Christopher G. Jones, “Test-Driven Development Goes to School”, Journal of Computing Sciences in Colleges, Volume 20, Issue 1, pg 220-231, 2004  Martin Fowler, “Continuous Integration”, http://www.martinfowler.com/articles/continuousIntegration.html, 2006http://www.martinfowler.com/articles/continuousIntegration.html 4/27/2011 12

13. “Well done. Here are the test results: You are a horrible person. I'm serious, that's what it says: A horrible person. We weren't even testing for that.” - GLaDOS Questions? 4/27/2011 13