Introduction ： ‘Skoll: Distributed Continuous Quality Assurance’ Morimichi Nishigaki

2 Background In-house QA –benefit fine level analyses of programs –shortcomings increased QA cost and schedule misleading results –difference from field

3 software trends Demand for user-specific customization –performance-intensive Severe cost and time-to-market pressures –limited resources for highly customizable software Distributed and evolution-oriented development processes –increase software churn rates, increase the need to detect, diagnose and fix faulty changes New Challenge: hundreds of options explosion of the software configuration space

4 Solution approach Skoll –general, continuous, feedback-driven process –around-the-world, around-the-clock QA. QA Process Subtask Central Collection Site results

5 Solutions Modeling the QA subtask configuration space –general representation with configuration options option settings inter-option constraints temporarily inter-option constrain Exploring the configuration space –search strategy based on uniform sampling –adaptation strategy for goal-driven process adaptation Feedback –automatic characterization, visualization Resource availability –scheduling adapting resource availability Process execution framework –Skoll process: provides framework to integrate QA techniques and tools

6 Skoll Project Developing and evaluating processes, methods, and support tools for distributed, continuous QA Skoll infrastructure: a set of components and services –issues: how tasks will be decomposed into subtasks how will they be implemented so they run on a very wide set of client platforms how will results be merged together and interpreted how should the process adapt to incoming results how will the results of the overall process be summarized

7 Skoll Infrastructure Configuration Space Model Intelligent Steering Agent Adaptation Strategies –Nearest neighbor –Temporary constraints –Terminate/modify subtasks Automatic characterization of subtask results Visualization Subtask Execution

8 Configuration Space Model Subtask: process parameterized by configuration options configuration: mappings option to a setting {(V i,C i )} Inter-option constraints: limitation of setting based on another setting (P i →P j ) example:

9 Intelligent Steering Agent (ISA) Control the global QA process –find valid configuration to allocate to client request –job configuration: package of QA subtask implementation application code configuration parameters build instructions QA specific code Planning problem –Blockbox planner initial state: base subtask configuration goal state: desired configuration, consistent with the end user –output: job configuration

10 Adaptation Strategies Results returned to ISA is ignored by default Monitor, analyze global state to modify future subtask assignments –Nearest neighbor test neighbor configurations of failed one –Temporary constraints temporary prevent further selection of job configuration –Terminate/modify subtasks monitors for situation where test program provide little new information switch resources to be spent unexecuted test program example: Nearest Neighbor

11 Automatic characterization of subtask results Used for –adapting the process –providing developers Classification Tree Analysis (CTA) 3 categories ERR-1 ERR-2 ERR-3 OK example: classification of 89 different configurations

12 Visualization Help developers organize and visualize results Server scoreboard manager –web-based query form

13 Subtask Execution Implementing QA subtasks run on client machine Server registration manager –web-based registration form, characterizing client platform –return ID and configuration template to client user-specific information, restriction by end user –Skoll client requests job configurations

14 Skoll process developers configuration model adaptation strategies generic QA subtask code ISA request Skoll client configuration template registration manager request job configuration job configuration results Adaptation server scoreboard manager query Automatic Characterization Skoll

15 Feasibility Study ACE+TAO –1MLOC+ –run on dozens of OS –hundreds of options –distributed ~140 developers

16 Study1: Clean Compilation Configuration model –subset of 17 binary-valued compile-time options –35 inter-option constraints Study execution –10 options and 7 constraints: 89 valid configurations –random sampling without replacement –No Nearest neighbor –60 of 89 failed Lessons –Iterative model building –useful automatic characterization

17 Study2: Testing with Default Runtime Options Configuration model –96 tests –test-specific options (one option per test ) Study execution –14 compile time options with 13 constrains –96 test-specific options with 120 constrains –29 configurations (build in Study 1 ) Results –compile: 98 failed 1979 success –tests: 152 failed 1827 success –discovered new bugs Lessons –easy extension and refinement –unreliability of configuration model and tree model –non-deterministic, single observation per valid configuration

18 Study3: Testing with Configurable Options Configuration model –14 compile options with 13 constrains –96 test-specific options with 120 constrains –6 multi-valued runtime options Study execution –18792 valid configuration –30 min. per test suites, 9,400 hours –Nearest neighbor Results –scalability –reconfirm: automatic characterization –adaptation

19 Summary Complex subtask configuration spaces are interactively modeled. Large scale QA processes are developed and executed on multiple clients. New real bugs in ACE+TAO were found. Automatic failure characterization simplified tracking down the causes.

20 END Thank you