Internal QA in Open Source Development Omer BarNir – MySQL AB omer@mysql.com

Agenda Background for this talk Open and Close Development Models Internal QA and Community Testing Q & A

Terminology QA Added Value The value of the test findings Closed / Traditional Development/Release/QA model Not the ‘Open Source’ model

Open and Closed Development Comparison Feature Complete Code Complete Development Testing Beta GA External Feedback External Testing Internal Testing Build Coding Bug Fixing Release Close Build Internal Testing External Testing and Feedback Alpha Beta RC GA Open

Meaning to Internal QA Close Source Open Source Most testing is done internally by QA ‘Large’ Testing Team Feedback from users is arriving late. QA is responsible to perform ‘User Scenarios’ and running tests in ‘real world’ configurations The added value of Internal QA is in performing tests and providing feedback about the quality of the product. Open Source Most testing is done externally (by the community). ‘Small’ Testing Team Feedback from users is arriving early and consistently. Large variety of real world ‘User Scenarios’ and platform configurations provided by community testing/ The added value of Internal QA needs to be defined

Community and Internal QA Testing Deterministic Tests follow a plan and coverage status is known Timing Testing Schedule is known and follow development needs. ‘Bugs’/’Features Test findings are compared to requirements / standards Control activity Monitoring, bug metrics, overall trend of issues, stabilizing the code Community Testing Not deterministic We don’t know if things are not used or are used without a problem. Random - No schedule to speak of. ‘Bugs’/’Features’ Is a given behavior the expected one (NIST tests found 20 issues with legacy functionality)

Putting It All Together Community Testing Internal QA Real World User Scenarios Configuration Variety Large Number of participants Determinism Time to Market Bug/Feature Clarity Monitoring Process Non Determinism Bug/Feature Ambiguity No Monitoring Limited User Scenarios Lab based Configurations Limited testing resources Strength Weakness Strength in one is weakness in the other.

Combining the Advantages of Both Approach: Positioning internal QA alongside community testing. Leveraging the best of both worlds by adjusting Internal QA’s work Testing Priorities Setting QA’s test priorities / schedules to bridge the gap between the desired development testing schedule and the testing coverage done by the community. Adjusting schedule of QA tasks based on feedback from Community Testing Focus Identifying features not likely to be tested by community Planning for internal QA to have more focus on these features.

Examples Testing Priorities Feature Focus V5.1 V5.0 Cluster Testing Partitioning Row Based Replication V5.0 Triggers / Views / Stored Procedures Feature Focus Cluster Testing Specific Integration of features (e.g. views & functions)

Similar QA Activities Monitoring: Release Assessments Bug Reviews Regression Testing Quality Procedures

Summary Differences & similarities in the release cycle Differences between Community testing and Internal QA How to position Internal QA to best leverage its added value alongside community testing.

Questions?