Quality Assurance in Sakai Past, Present and Future Pete Peterson, Sakai Foundation Megan May, Indiana University

10th Sakai Conference - Boston, MA, U.S.A. Agenda What is QA? Past Present Future Discussion/Questions? July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. What is QA/QC? Quality Assurance (QA) is the process of verifying or determining whether products or services meet or exceed customer expectations. QA considers all aspects of design, development, production, and service. When implemented correctly, QA provides a continuous and consistent improvement of processes within a product lifecycle. Quality Control (QC), a subset of QA, is employed to ensure a certain level of quality in a product or service. The basic goal of QC is to ensure that the products, services, or processes provided meet specific requirements. Often, Quality Control is confused with Quality Assurance. Though the two are very similar, there are some basic differences. QC is concerned with the product, while QA is process–oriented. Simply put, QA ensures that a product or service is developed and implemented in the right way; while QC evaluates whether or not the end result is satisfactory. July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. What is QA/QC? There are three essential questions that need to be answered and tested long before a product is placed into production, and QA/QC activities focus on answering these questions: Does it do what we expect it to do? Does it do what the end-user expects it to do? Does it do “a” and “b” correctly and consistently? July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Key Points of QA/QC Quality is not free: There are costs associated with implementing these additional steps into your projects and development processes—time, money, resources, etc. These costs, however, are small in comparison to the benefits of having increased end-user satisfaction, reduced support requirements, and functionally accepted applications and systems. Repeatable: The QA/QC process should be repeatable and flexible, so it can be easily applied to other projects as they arise. July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Key Points of QA/QC Comprehensive: QA/QC should be included in the project from the beginning to the end. The goal of this participation is to examine the project from all angles. This not only allows the QA/QC group to create better tests, but also to provide feedback and a global perspective to the developer community. July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Key Points of QA/QC Ongoing: QA/QC doesn’t stop once a project is complete; in fact this is where much of the QA is applied. Examining pitfalls and past mistakes can improve design, development and testing, as can documenting procedural successes and areas that need improvement. July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Key Points of QA/QC Autonomous: It is essential for the QA/QC group to have a high level of autonomy. Often the QA/QC process is performed in-house by the developers or project managers themselves. Although well-intentioned, this approach is almost always ill-advised and precarious. It is difficult to remove oneself, step back and see things from the diverse perspectives needed for an effective QA/QC analysis. July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Past “Those who don't know the [past] are destined to repeat it.” Edmund Burke July 2009 10th Sakai Conference - Boston, MA, U.S.A.

Sakai’s Testing & Release Management Process Primarily At end of the cycle. Mostly manual process great emphasis on participation of RM. A lot of the release/product management is bleeds into the QA process.

Formulation of QA Network Amsterdam Boston Cambridge Cape Town Columbia Indiana rSmart Charles Sturt Georgia Tech 3 servers at MIT (mysql, oracle, hsdb) There are 13-15 QA servers hosted at variety of institutions. Deployment testing on wider variety of platforms Tested conversation scripts by retaining data. Recent additions of Nagoya, Marist & UMich

Past QA Workgroup Participation 2.2.1 23 People 10 Institutions 4 Countries 2.3.0 44 People 17 Institutions 5 Countries 2.4.0 93 People 37 Institutions 11 Countries 2.5.0 56 People 17 Institutions 4 Countries - The QA WG is also a community of volunteers. There is a core group in the WG testing - During these releases high degree of emphasis placed on bug verification (~85%)

10th Sakai Conference - Boston, MA, U.S.A. Redefining processes Defined milestones http://confluence.sakaiproject.org/x/VwAiAQ No new features in maintenance releases Better oversight of merges into maintenance release branches July 2009 10th Sakai Conference - Boston, MA, U.S.A.

Other Accomplishments Expansion of test script library Raised awareness of the importance of unit and performance testing throughout community Availability of Static Code review Integration of Accessibility WG activities in general QA cycle Inclusion of 3rd party licensing review in QA process July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Sakai QA - Present July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Sakai QA - Present Finalizing Sakai 2.6 release Finalizing Sakai 2.5 release So everything is done right? July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Sakai QA - Present Sakai 2.6.1 and 2.5.6 coming soon to a QA server near you Testing for maintenance branch merges Updates of test plan July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Sakai QA - Present The Links Slide http://www.sakaiproject.org/portal http://jira.sakaiproject.org/ http://confluence.sakaiproject.org/ QA http://confluence.sakaiproject.org/display/QA/Home Release Management http://confluence.sakaiproject.org/display/REL/Release+Management+Working+Group July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Resources K2 Sakai 3 Sakai 2.5 K1 Sakai 2.6 Sakai 2.7 July 2009 10th Sakai Conference - Boston, MA, U.S.A.

10th Sakai Conference - Boston, MA, U.S.A. Future Automation Inclusion of test cases in every jira Continuous testing July 2009 10th Sakai Conference - Boston, MA, U.S.A.