1. Continuous QA Sewit Adams (Colorado State University) Bin Gao (Michigan State University) Jerry Neal (Indiana University)

2. Topics Overview of Project Testing KFS Testing Methodology The Test Environment Lessons Learned Test Case Examples

3. Objective

4. OVERVIEW OF PROJECT TESTING

5. Testing Definitions(1) Unit Test (IEEE Definition): A set of one or more computer program modules together with associated control data, usage procedures, and operating procedures. Integration Test: A test that groups modules together to test their interactions. Conformance-Directed Testing: Intent of testing is to demonstrate conformance to required capabilities.

6. Testing Definitions(2) Unit Test:

7. Testing Definitions(3) Integration Test:

8. Example Test Definition (1)

9. Example Test Definition (2)

10. Why Unit Test Exposes problems earlier in development cycle. Encourages Refactoring Forces Regression Testing

11. KUALI TESTING METHODOLOGY

12. Writing Unit Tests (1) Who writes unit tests –API’s owner/implementer –Pair programmer When to write tests –Test first –Code first What are the criteria –functional specifications –APIs

13. Writing Unit Tests (2) What are unit tested –Service and utility methods –Application configuration What may not be unit tested –Third party libraries –Database Follow three As pattern –Assemble: prepare test data –Act: execute a tested method against the data –Assert: compare the test results with the expected

14. Writing Unit Tests (3) How to prepare test data? –Well-maintained data set –Test fixture –Properties file –Mock objects How many tests for each method? –Cover all boundary conditions

15. Kuali Test Framework (1) Objectives –Ease the unit test writing –Make the unit tests maintainable Implementation –org.kuali.kfs.context.KualiTestBase –Extends junit.framework.TestCase –overrides TestCase#runbare() Test extends KualiTestBase

16. Kuali Test Framework (2) Features –Spring context startup –Transaction management –Authorization management –Unsolved issue Annotations –@ConfigureContext –@RelatesTo

17. Kuali Test Framework (3) Using Kuali test framework public class ServiceTest extends KualiTestBase { @ConfigureContext(session = KULUSER) public void testCreate() throws Exception{ // assembly – act - assert } @ RelatesTo(KULRNE42) public void testRoute() throws Exception{ // assembly – act - assert }

18. THE TEST ENVIRONMENT

19. Test Package Structure Parallel Package Structure – work/src/org/kuali/module/someModule – test/src/org/kuali/module/someModule

20. How To Run Unit Test (1)

21. How To Run Unit Test (2)

22. How To Run Unit Test (3)

23. How To Run Unit Test (4) Can also run tests using ant target.

24. Continuous Integration With Anthill

25. LESSONS LEARNED

26. Test Code Standards (1) Leads to easier maintenance Documents code use better Test code is not second class!

27. Follow normal coding best practices Establish test code specific standards –Test case naming conventions –Best Use of assert statements –Document test case (what is it testing?) –Avoid hard coding data –Keep test methods short (one “Act”) Test Code Standards (2)

28. Code Coverage (1) In Theory, should test all services Or even better, all public methods But, we must meet deadlines! Therefore, we must decide which tests are more important to write

30. Want tests with high value but low maintenance Determine V/M rating for possible tests Start with tests that bring the most reward As time allows, move on to tests that reward less Code Coverage (2)

31. Establish value of test with a simple rating system like low, medium, high or 1-10 Possible criteria for determining value: –Importance of functionality being tested –Prevents reoccurrence of a bug? –Usefulness of documentation from test case Code Coverage (3)

32. Establish maintenance rating with similar system. Lower ratings are preferred. Possible criteria for determining maint. rating: –Dependencies (service, external systems, …) –Fragility of test data –Test state setup (Number of setup calls, services used, mock objects needed, …) –Time required to write test Code Coverage (4)

33. Duplicate Tests Cause more overhead when refactoring Provide no additional value More developer time when a failure occurs Duplication easily masked: –Testing method that encapsulates already tested methods (indirect testing) –Multiple tests with different parameters that are part of an Equivalence Class

34. Continuous Build (1) Frequent builds (every 15 minutes or each check-in) help identify and correct test failures more quickly. Important to keep tests all passing at high rate: –Pinpointing new failures easier –Better chance developer who caused failure will fix the test Suites and Package Tests help identify failures before build

35. Overseen by QA Manager Notices sent out when failures occur Jira assignments made for tests not resolved after one day Developers encouraged to spend some time early in day resolving test issues Continuous Build (2)

36. Sharing Knowledge Establish ways for sharing test related information –Testing KB –External Resources –Test Reviews Document test failures –CVS comments –Jira

37. Beyond Unit Tests Difficult to test web layer Many web bugs reoccur multiple times from framework modifications Need mechanism for doing system tests Developing Http Unit tests

38. Continuous QA Test standards continue to evolve More test recipes are developed Testing KB grows More test utilities, base methods, and improvements in data handling and dependencies Better ability to identify tests to run before check-in

39. QUESTIONS?