1. What does QA mean to Sakai? Megan May – Sakai Foundation Aaron Zeckoski – CARET Alan Berg – UVA David Horwitz – Cape Town Seth Theriault- Columbia Linda Place – U of M

2. 2 Today’s Session What is QA? Unit/integration testing Static code review Release/Branch management QA server network QA WG Performance testing

3. 3 What is quality assurance? What is the role of the QA Director? the set of planned and systematic actions necessary to provide appropriate confidence that a product or service will satisfy the requirements for quality.

4. Unit/Integration Tests Aaron Zeckoski

5. 5 Test Driven Development Create the Class/API -> Write the Test -> Program the Implementation -> Run the Test Forces you to use your own method (and hopefully check if it is intuitive to use) Requires an immediate check against the javadoc (API) Makes the developer think about how the method will work and what it does BEFORE they write any code http://en.wikipedia.org/wiki/Test-driven_development

6. 6 Bug fix test cases When there is an issue created against your code you should create a test case to test for that specific failure before you fix it These steps –Create a test case which fails as indicated in the issue from the issue tracker (JIRA) –Reference the JIRA URL in the comment for the test case –Run the test and make sure it fails –Fix the problem –Rerun the test to make sure it passes –Now you are safer from regressions

7. 7 Tests vs. Code Review Static code review is valuable and helpful Peer code review is more valuable Neither is a substitute for unit testing and integration testing though Why? –They do not provide for protection from regression –They do not supply the context that tests do Tests show how to use the code and provide a type of documentation

8. Static Code Review Alan Berg

9. 9 Static Code Review Functional testing –Guarantees a minimum quality –Black box looks from the outside in –Does not cover all the code –Labor intensive –Does not totally scale as the code base scales

10. 10 Static Code Review Static Code Reviews –Looks at the source code not a running instance –White box, looks from the inside out –Looks through all the code base once a day –Automated –Steps over commit boundaries –Information overload –Requires the right mentality from the community –Need to eat the bugs with little fuss

11. 11 TOUR qa1-nl.sakaiproject.org/codereview/bug_dashboard/index_generic.html

12. 12 TOUR qa1-nl.sakaiproject.org/codereview/bug_dashboard/index_generic.html

13. 13 Source code 13

14. 14 Evolutionary 14 Using PMD, Findbugs, QJ Pro, Jlint and custom Perl code Tools getting better Findbugs does not have many False positives PMD has configurable rules More shaping the information to avoid information overload Engaging in debate on how to deal with bugs Need to change developers perceptions

15. 15 Other 15 Code coverage Automatic removal of simple bug types script Tracking of long term use of memory Provisioning for performance testing More to come

16. Branch/Release Management

17. 17 Branch Management & Enlightened Self interest Why would we want to be branch managers? –first to go to 2.5.x –wanted a say in ensuring a high quality branch for production –Its not actually a big overhead outside of the release cycle

18. QA Server Network Seth Theriault David Horwitz

19. 19 QA Server Network Amsterdam Boston U Cambridge Cape Town Columbia Indiana rSmart Georgia Tech Charles Sturt U Coming soon - server in Japan

20. 20 Running a qa server What you need: –a computer –a network connection –a database –know how to compile and install Sakai Initial setup –Build server

21. 21 Running a qa server Updating and maintenance –downloading source –build deploy –upgrade data (maybe)‏ –usually once a week during release

22. QA WG

23. 23 Collab https://collab.sakaiproject.org/portal JIRA Confluence http://jira.sakaiproject.org/ http://confluence.sakaiproject.org/confluence/displ ay/QA/Home WG Tools

24. Sakai Performance Testing Current model and limitations Desired model and benefits Dr. Linda M Place, University of Michigan

25. 25 Talking Points Current model –Black box testing –Performance failure debugging and fixing Desired model –White box testing –White box testing with baseline load –Minimize production performance failures

26. 26 Black Box Testing Black box testing –Running system with projected use-case scenarios for acceptable throughput and response times How will users experience application? –Baseline comparison Throughput and response times compared against previous version test results Looking for as good as or better results from new version –Typically occurs at end of QA process

27. 27 Qualities Being Tested Push application load against a system to identify quality of performance –Transaction response time for end users –Throughput (hits per second) –Reliability (percentage of failed transactions) –Scalability (what happens when scale increases significantly) –Capacity (upper limits of system)

28. 28 Results Are metrics being met? –Measures of success Current Sakai test metrics –Baseline or better response times and throughput –New transactions do not disrupt baseline metrics –New transaction stress tests How many occurrences before system becomes unusable?

29. 29 Failed Performance Debugging Designing isolated tests to address obvious performance problems in production code –Use tests to isolate performance cause –Use tests to verify fix before adding to production instance Putting out fires

30. 30 White Box Testing Pushing system to identify application, database, operating system, or network problems –Tune environment to identify and address specific problems –Tests watched by developers, DBA, system and network administrators, and performance engineers Typically occurs during code development

31. 31 Little in Sakai Today Developers may design performance tests as part of development –Process not codified as required standard for development –Most developers do not have a system of sufficient scale to push code adequately –All components of production environment not used and monitored Developers; database, system & network administrators; performance engineer

32. 32 Value to Sakai Adding white box testing into Sakai QA process finds and fixes numerous performance problems before they reach formal QA process Engages Operations team early in development cycle rather than after performance problem affects user community Debugging easier prior to full integration with application suite

33. 33 Want White Box Testing? Stay in room for next session –Sakai Community Performance Testing: A Working Proof of Concept Participate in community model to improve your code prior to subjecting it to QA Don’t expect someone else to identify and fix your codes performance shortcomings

34. 34 Questions