1. 8th Sakai Conference4-7 December 2007 Newport Beach What does Quality Assurance Mean to Sakai? Alan Berg Megan May Seth Theriault

2. Topics What does a QA Director do? QA Today –Who is in the WG –Types of testing currently done and release activities –Strides forward Resources available –QA Network 2.5 Release Activities Update QA Tomorrow Questions

3. Role of QA Director Coordination! –Help bring parties interested in the same testing efforts together –Foster initiatives that support quality throughout the community –Assist in bridging the gap between test and design & development teams –Ensure release activities are transparent

4. QA Today Development Testing & Software Release Requirement Gathering and Design Focus on Functional Testing

5. Who is the QA WG? 2.4.0 93 People 37 Institutions 11 Countries 2.2.1 23 People 10 Institutions 4 Countries 2.3.0 44 People 17 Institutions 5 Countries 2.5.0 56 People 17 Institutions 4 Countries... How do they get involved?

6. Strides Forward Deployment testing on wider variety of platforms Integration of Accessibility WG activities in general QA cycle Expanding/updating library of test scripts Reflection on Release process –http://confluence.sakaiproject.org/confluence/x/fbUhttp://confluence.sakaiproject.org/confluence/x/fbU

7. QA Network Amsterdam Boston U Cambridge Cape Town Columbia Indiana rSmart Georgia Tech Coming soon servers in Japan and Australia

8. Collab How to be in the know! https://collab.sakaiproject.org/portal

9. Jira http://jira.sakaiproject.org/

10. Confluence http://confluence.sakaiproject.org/confluence/display/QA/Home

11. Perceived Quality Level

12. Quality, like beauty, is clearly in the eyes of the beholder.

13. Perceived Quality Poll

14. QA Tomorrow Sakai Community Concerns –Several prominent campuses experience serious performance problems with 2.4 2.5 Release Update –Further refinement of process –Goals for 2.5

15. Paths Forward Common set of testing vocabulary –Unit Testing –Integration testing –System Testing –Performance Load testing ( lmp@umich.edu ) lmp@umich.edu Integration testing Development Testing & Software Release Requirement Gathering and Design http://confluence.sakaiproject.org/confluence/x/M4B5

16. Roadmap for Tomorrow

17. Metrics to Describe Quality 2.3.0 – 85% Verification 2.2.0 – 85% Verification (1,345 JIRA Tickets) (611 JIRA Tickets) (504 JIRA Tickets) 2.4.0 – 82% Verification 2.5.0 – Past bug verification

18. Metrics to Describe Quality

19. Static code Review Search for bug patterns What to process: Source code, Binary Two types of bug pattern –Generic Classic (buffer overflow, input validation) –Contextual Language specific Framework specific Configuration specific Proximity effect

20. Hinting nightly http://qa1-nl.sakaiproject.org

21. Random Example

22. Duplicate Code

23. Do we wish to use the reports more aggressively? Are the developers taking note? –Bugs reported are not always correct –Bugs found not always important –But duplicate code, unit test coverage, and failing to deal properly with exceptions hint strongly at quality. –Not enough Contextual searching for Sakai and spring configuration specific issues –Reports do not explain enough –Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.

24. A year on Update prototype –Less noise more explanation –Graphical tracking of change Recommend Eclipse Plugins Look at emerging products See as part of a series of QA processes. Agree on metrics to act on and then act.

25. A good book to read Secure Programming with Static Analysis –Brian Chess, Jacob West Addison Wesley ISBN: 9780321424778 Includes CD with software and examples.

26. What you can do Pledge Resources –What happens in the community has a direct local impact –Easy point of entry into Sakai –All skill sets are needed Discussion –Why hasn’t this been a priority? –What are obstacles in the way and how do we remove them?

28. Questions

29. Other Sessions of Interest 2.5 QA Workshop (Thursday, 13:00 - 14:30 in Salon 5) The Automated Testing Conundrum in Sakai (Thursday, 15:10 - 16:10 in Salon 5)