The complex world of Quality Assurance on OpenOffice.org Thorsten Ziehm Manager Quality Assurance for Ooo/SO at Sun Microsystems

Agenda Who is Thorsten Ziehm Quality Assurance for Software The QA processes for OOo The QA project on OOo Future Resume

Who is Thorsten Ziehm ● I work for StarOffice since 1996 ● I started as automated tester to test StarOffice with QA-Partners (Segue/Borland) until SO changed to VCL ● I worked with development to make SO/OOo testable with VCL Testtool ● Since Oct I am the manager of the whole QA team for SO and OOo at Sun Microsystems => nearly 13 years of experience in QA for OOo

Quality Assurance for Software ISO standard for Software Testing V-Model Statements

QA for Software ISO standard 9126 ISO standard 9126 ● Functionality A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. ● Reliability A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. ● Usability A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users. ● Efficiency A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions. ● Maintainability A set of attributes that bear on the effort needed to make specified modifications. ● Portability A set of attributes that bear on the ability of software to be transferred from one environment to another.

QA for Software The V-Model The V-Model Business Case / Requirements Release / Acceptance Testing Construct Component System Specification System Design Component DesignComponent Testing Interface Testing System Testing Construction & Integration Validation Verification API/URE/SDK UNIT-Tests Complex test QA Community Customer, User, Community The V-Model (or VEE model) is a systems development model designed to simplify the understanding of the complexity associated with developing systems.

QA for Software Testing is not Debugging The goal of debugging is to find and fix the defect or error condition. The function of testing is to find systematically and targeted the results of the errors. Software systems are not error free Currently and in near future an error free software system is not possible, when the system has an higher level of complexity and amount of lines of code. Error free code is not reachable with testing When all test cases do not show any error, you could not be save, that new test cases or test scenarios show new errors. Testing does not bring in the quality Testing can identify errors and defects only. The quality of the system is based on the developed code. Links for Software Testing and Quality Assurance for Software International Software Testing Qualification Board International Software Testing Qualification Board (ISTQB)ISTQB WikiPedia entry for Software Testing

QA processes working on Child Work Spaces Sun internal processes only automated testing on CWS automated testing on Master manual QA / Testing

QA processes ● Testing on Child Work Spaces (CWS) ● bring in quality approved code on master code line ● code changes in separate areas ● concentrate on defined test cases (separate areas) ● QA approval can be given easier and quicker ● Information/Links : – What is a Child Work Space What is a Child Work Space – CWS policies CWS policies – work flow for approval of a CWS work flow for approval of a CWS ● (instead of integrating and testing all code on master only)

QA processes (Sun internal processes only) ● API and complex tests for each CWS (Sun internal) ● Performance tests (load/save) for each CWS (Sun internal) ● performance tests on start-up (with system reboot) for Linux on each master build (Sun internal) ● ConvWatch for CWS with changes/fixes in filter (Sun internal) ● check crash reporter data base on regular base ● Valgrind* tests on regular base (Sun internal) Valgrind *

QA processes - in QA community ● Automated testing with TestTool on each CWS ● “required tests” (set of 26 tests) are required for QA approval for all CWS ● additional tests can/should be selected by the QA responsible for the CWS ● QA approval for CWS only, when all tests show same result as in corresponding Master build ● the automated test scripts should show easily and quickly general broken areas and regressions in special areas ● an automated process has to need less than 1 night

QA processes - in QA community ● Automated testing with TestTool on Master ● for each master the full set of test scripts should run (currently only each 3 rd builds all tests run) ● create a basis for comparison the results of CWS testing ● find regressions in Master easily and quickly which aren't in a CWS ● run automated tests on all platforms regularly ● the automated process has to need less than one build cycle (~5 days in general)

QA Processes Child Work Space Manual Testing Required API tests BuildBots Performanc e Tests Load/Save 26 required automated tests Manual Testing Communit y Su n Master Build Manual Testing Communit y all automated tests Manual Testing Su n

The QA project The QA community What is their job? What isn't their job? What is needed?

The QA community ● Full time employees at Sun ● The QA teams in the QA project on OOo ● The L10N projects ● The general user/customer of OOo ● The developer at/with OOo

The QA community What is their job? ● Working in iTeams* (e.g. for new features) ● identifying requirements and business cases ● working on system or functional specification ● design test cases based on the functional specification ● testing the implementation with designed test cases ● Working on Defects ● reproduce bug reports ● finding defects and errors in old/new functionality ● verifying fixed issues in CWS and Master ●... * iTeam : a team which is responsible for implement/integrate new functionality/features

The QA community What is not their job? ● design the system and the component for new integrations ● write the code ● debugging the error or defect ● writing unit tests ● writing API or complex tests ● make code analysis ● analyse performance issues ● etc.

The QA community What is needed! ● a powerful lead for the QA project ● responsible team members in the QA teams ● active QA community members in different L10N communities ● automated QA processes outside Sun ● easier and quicker release processes for all languages ● one tool for test cases / one defined way for testing inside the QA project

Future/Wishes TestBotsQA Portal All CWS will be checked automatically with a set of automated test scripts and/or other automated QA processes Benefit : General stability and regressions in general functionality can be checked easily by/for everybody All QA relevant data should be stored for a CWS and the master builds on a general available portal on OOo – QUASTE!? (Results of API-Tests, Automated Tests, Performance, BuildBots …)

Resume ● quality is a compromise between time and resources ● it isn't possible to get 'all' bugs fixed – the issue is too complex or too simple – there isn't QA person who can check/verify the issue – there isn't a developer who can fix the issue – there isn't a UX person who can design the feature ● often there isn't the time to get all the issues fixed in time => pressure is high for all distros ● more show-stopper issues need to be accepted to get a well qualified product => but the time is running away... often

Meet the Sun Experts at the Exhibition

Appendix / Links Who is Thorsten Ziehm : m (German) ISO 9126 (ISO standard for Quality on Software): 6 V-Model : model International Software Testing Qualification Board (ISTQB) : d / g General Information (wiki entry) about Software Testing : Important information for CWS on OOo : Child Work Space Handling : CWS Policies : QA approval process for CWS : Information about QUASTE : e /