1. Automated Testing in Sakai Testing applications and services in isolation and in context Josh Holtzman, UC Berkeley David Haines, University of Michigan

2. Goals Discuss developer-centric testing in Sakai –Purposes –Concepts –Tools Demonstrate a simple running example of application and service testing

3. Why Formal Testing? Ensures that you know what the code is supposed to do Ensures that code does what it should do Ensures that the code doesn’t regress when changes are introduced Developer testing is not to avoid QA

4. Types of Testing Object (Logic/ Unit ) Testing Integration Testing Functional Testing System Testing Load Testing (Capacity, Stress, Performance)

5. Load Testing (Ops) Does it perform realistically? –Capacity – Can it deal with X users / transactions per hour? –Stress – What happens when it gets overloaded? –Performance – What model describes the behavior of the system? –Tools - Loadrunner, jmeter, grinder

6. System Testing (QA) Can the entire system perform the required task? –Integrates the full software stack, including the web server –Tools include httpunit, jmeter, jwebunit, people, others

7. Functional Testing (Dev, QA) Test specific behavior in deployed system Can a user really add a new site? Slowest method of developer testing Tools –Selenium-IDE, httpunit, jwebunit

8. Integration Testing (Dev) Tests an object with a real environment –Uses a test harness to bootstrap the application’s environment –Test the object using true collaborators –Slow. Must start up the entire environment –Not a replacement for logic testing! Tools –Sakai test harness

9. Logic/Object/ Unit Testing (Dev) Test object behavior Fastest kind of developer testing Create an object, setup its local environment, call a method, and verify its result is what as expected –Tests a single unit of code at a time –Does not test an object in a runtime environment –Must simulate collaborators as needed –Tests can and should be written before the implementation –Write a failing test first!

10. Logic Testing (continued) Tools –Behavior - Junit –Runtime Environment Mock objects can help with collaborators (jmock, easymock) Stubs can help with collaborators (but get messy quickly) Dynamic proxies –Ecology - Coverage tools help assess how much code has not been tested (jcoverage, clover)

11. Logic Testing (continued) JUnit Architecture –Test method –Test Cases –Test Suites

12. Types of Developer Testing Logic –Does this logic work correctly? Integration –Does it talk nice? Functional –Demonstrate that works in a deployment.

13. Logic Testing Demo Test –Test a single assertion TestCase –Collection of related tests –Uses fixtures for setup and teardown TestSuite –Collection of TestCases –Uses fixture for one time setup and teardown

14. Integration Testing Add real service and test to make sure data gets through

15. Functional Testing Does it work in a deployed system? Get it up and deployed and running. Click through and make sure it works.

16. Summary Anything not tested before production gets tested in production Fail first Test what you are writing, not the whole system Changing code to be able to test is often necessary Refactoring is good

17. Resources JUnit –http://junit.org/http://junit.org/ –http://junit.sourceforge.net/doc/cookbook/cookbook.htmhttp://junit.sourceforge.net/doc/cookbook/cookbook.htm Mock objects –http://www.mockobjects.comhttp://www.mockobjects.com –http://www-128.ibm.com/developerworks/library/j-mocktest.htmlhttp://www-128.ibm.com/developerworks/library/j-mocktest.html Test harness –https://source.sakaiproject.org/svn/test-harness/trunk/xdocs/README-INTEGRATION-TESTING.txthttps://source.sakaiproject.org/svn/test-harness/trunk/xdocs/README-INTEGRATION-TESTING.txt Sample code –https://source.sakaiproject.org/playground/trunk/unittesttalk/hello-world/https://source.sakaiproject.org/playground/trunk/unittesttalk/hello-world/