1. QA Process within OEM Services Ethan Chang QA Engineer OEM Service, Canonical ethan.chang@canonical.com

2. 2 | Internal use only Agenda Overview Process Flow Bug management Documents Checkbox Introduction

3. 3 | Internal use only Quality Advantages Three QA passes – Upstream QA – Platform Community Beta QA – OEM Services QA Many “Eyes” on Ubuntu – Over 100 bugs a day submitted against Ubuntu – Over 1300 unique Ubuntu mailing list posts per day – Over 300,000 bugs logged to date – Over 100,000 bugs fixed released

4. 4 | Internal use only QA Team Overview Canonical QA – Driven by a core of QA Professionals in Canonical – Community supported with a network of thousands – Beta user testing begins as soon as the repos are created OEM Service QA – Designed around more formal SQA common practices – Globally distributed with members in USA, Spain, France, Taiwan, and Mainland China to maximize “up time”. Tools – All tools used by Canonical QA and Community QA are open source and all test cases are public. – Automated testing combined with manual repeatable tests and Beta usability testing.

5. 5 | Internal use only Actual Ubuntu Testing Workflow

6. 6 | Internal use only OEM QA Workflow

7. 7 | Internal use only Defect Workflow

8. 8 | Internal use only OEM QA Case Structure ● Creation cases based on customer requirements ● Creation of device specific cases ● Integration of cases provided by customer

9. 9 | Internal use only Test Case Creation and Maintenance Open source Contributed to by community members and reviewed by Canonical Constantly growing suite of cases

10. 10 | Internal use only Documents OEM Services QA has produced a series of documents that provide more detailed information on our process. – Test Planning – Test Reporting – Test Cases – Testing SOP

11. 11 | Internal use only Documents – Test Planning System requirements Application stack Canonical deliverables defined Milestone criteria Bug severity definitions Bug status definitions Test case definitions

12. 12 | Internal use only Documents – Test Cases Version controlled Results tracking across releases Simultaneous edits Available from any browser

13. 13 | Internal use only Documents – Testing SOP Checkbox instruction How to submit result to Certification webstie guide Launchpad usage guide

14. 14 | Internal use only Checkbox An easy tool for system testing Automated and manual test support Guided testing Plugin Architecture Simple report interface Online results publishing

15. 15 | Internal use only Checkbox - an easy tool for system testing Easy installation Easy customization Easy development Easy distribution

16. 16 | Internal use only Checkbox - Automated and manual test support Simply click on “Test” button to start testing Just one click

17. 17 | Internal use only Checkbox - Simple report interface Directly report bug to Launchpad Checkbox will collect all the logs

18. 18 | Internal use only Checkbox - Online results publishing Input unique Secure ID to submit result to Canonical Certification website

19. 19 | Internal use only Checkbox - Easy to review Test Report XML format Review by any browser Review before submit

20. 20 | Internal use only Results Tracking

21. 21 | Internal use only Checkbox - Urwid interface for Sever testing Install Checkbox-urwid to test server system

22. 22 | Internal use only Apport Automated crash detection Automated collection of system information Automated collection of error state Automated submission of defects with collected information attached

23. 23 | Internal use only Linux Desktop Testing Project (LDTP) Full desktop automation suite Integrated with Gnome Desktop Python based scripting Open Source

24. 24 | Internal use only Q&A

25. 25 | Internal use only Thank you for your time