LOGO Evaluating Quality-in-Use Using Bayesian Networks M.A. Moraga 1, M.F. Bertoa 2, M.C. Morcillo 2, C. Calero 1, A. Vallecillo 2 1 Universidad de Castilla-La Mancha 2 Universidad de Málaga (Spain)

LOGO Paphos, Cyprus - July 8 th, 2008 Contents Introduction 1 ISO/IEC Bayesian Belief Networks 3 BNs for evaluating QiU 4 Conclusions 5

LOGO Paphos, Cyprus - July 8 th, 2008 Introduction. The ISO approach  Assessing the quality of a software product is, in general, a complex and difficult task.  ISO/IEC 9126 proposes three levels of quality:  Internal quality uses a “white box” view of software products  External quality uses a “black box” view  Quality in use (QiU) is the end-user viewpoint when the software product is used in its working environment

LOGO Paphos, Cyprus - July 8 th, 2008 ISO/IEC 9126 Quality Efficiency Maintainability Portability Functionality Reliability Usability Efficiency Maintainability Portability Functionality Reliability Usability QiU Internal External Productivity Safety Satisfaction Effectiveness

LOGO Paphos, Cyprus - July 8 th, 2008 ISO current approach is questionable  Most efforts so far have been put on evaluating the internal and external quality  … based on the assumption that a good external quality would guarantee a good quality-in-use (QiU)  But we all know that quality does not work that way!  The QiU should be the driving force!

LOGO Paphos, Cyprus - July 8 th, 2008 Reasons to challenge the traditional approach 1Not all the external quality characteristics of a software product have the same influence on its QiU 2The direct dependency between the EQ and the QiU, forces to over-specify some of the product aspects (which are non critical for the end user), for the sake of ensuring a certain level of QiU.  Unnecessarily increasing costs and development efforts, without a direct effect on the advantages that the end-user perceives 3Users normally tend to perceive as equally important all the external quality characteristics of a software product when asked about them in the abstract  However, when the context-of-use of a soft-ware product is fixed, users are able to tell apart the product characteristics that really matter (from their point view), from those that are desirable, but not critical

LOGO Paphos, Cyprus - July 8 th, 2008 Our position in this paper  We change the focus of the quality assessment  concentrating on the QiU as the driving factor for designing a software product, or for selecting the product that better fits a user’s needs  We propose a “backwards” analysis of the relationship between the external quality and the QiU  The analysis tries to determine the external quality sub- characteristics that are really relevant to ensure the required level of quality in a given context of use  in order to avoid superfluous costs or irrelevant features – which may unnecessarily in-crease the final price of the product  We propose Bayesian Belief Networks to model such relationships, and propose a method to build them for different contexts of use.

LOGO Paphos, Cyprus - July 8 th, 2008 Contents Introduction 1 ISO/IEC Bayesian Belief Networks 3 BNs for evaluating QiU 4 Conclusions 5

LOGO Paphos, Cyprus - July 8 th, 2008 Bayesian Belief Networks  A BN is a directed acyclic graph, whose nodes are the uncertain variables and the edges are the casual or influential links between variables. The different relationships between the characteristics and sub-characteristics of the external quality and the QiU Using BNs We can model The degree of dependence or influence between them

LOGO Paphos, Cyprus - July 8 th, 2008 An example of a Bayesian Network [Taken from Fenton and Neil “Software Metrics Roadmap”]

LOGO Paphos, Cyprus - July 8 th, 2008 An example of a Bayesian Network [Taken from Fenton and Neil “Software Metrics Roadmap”]

LOGO Paphos, Cyprus - July 8 th, 2008 Bayesian Belief Networks Steps Make inferences about the values of the variables Define the conditional probability tables Define the structure Train the BN through a set of controlled experiments

LOGO Paphos, Cyprus - July 8 th, 2008 Contents Introduction 1 ISO/IEC Bayesian Belief Networks 3 BNs for evaluating QiU 4 Conclusions 5

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU HYPOTHESIS EQ has influence on the QiU and this influence can be modelled and studied through a BN We can conduct backwards analysis on the required level of EQ to ensure a given level of QiU

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1) 3) Link 2)

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1 st Approach  Study the relationships between the different characteristics, identifying the EQ characteristics that have a significant influence on the QiU characteristics. Quality in Use SafetySatisfactionProductivityEffectiveness External Quality Functionality XXXX Reliability XXX Usability XX Efficiency XX Portability X Maintainability X

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1 st Approach  BN

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1 st Approach  Probability tables:  3 categories (IEEE Std. 1061):Acceptable, Marginal and Unacceptable  The tables of the upper level nodes have a single entry with the evaluation of the external quality sub- characteristics (opinion of experts).  The probability tables of the nodes in the next level have been defined taking into account the degree of influence that each one has on the corresponding QiU characteristic.

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1 st Approach

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 1 st Approach  Problem: all sub-characteristics of an external quality characteristic have the same influence on all related QiU characteristics Is Suitability as “important” for Safety as for Satisfaction? The answer is, of course, No. 2 nd Approach

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 2 nd Approach  Define relationships between EQ sub-characteristics and QiU characteristics

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 2 nd Approach  BN

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 2 nd Approach  Problem: it produces a very high number of entries on the nodes that represent the QiU characteristics.  And therefore, the definition of the probability tables is very laborious and cumbersome.  Solution: the introduction of synthetic nodes 3 rd Approach

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 3 rd Approach  A synthetic node among sub-characteristics of each EQ characteristic and each QiU characteristic.

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU 3 rd Approach  Advantages:  It reduces the number of entries in the probability tables.  It avoids the undesirable influence of EQ sub-characteristics on the QiU characteristics when they do not have a direct relationship This network achieved a finer granularity while avoiding inconsistent relations  Disadvantages:  the concept of external quality characteristic is lost.  However, we could easily create four individual BNs, one for each QiU characteristic, and then combine them to form the complete Bayesian Network.

LOGO Paphos, Cyprus - July 8 th, 2008 BNs for evaluating QiU Each QiU characteristic can be studied independently

LOGO Paphos, Cyprus - July 8 th, 2008 Contents Introduction 1 ISO/IEC Bayesian Belief Networks 3 BNs for evaluating QiU 4 Conclusions 5

LOGO Paphos, Cyprus - July 8 th, 2008 Conclusions  We have proposed to change the focus of the Evaluation of the Overall Quality of a Software Product  Evaluation should be focused on the QiU – and then use a backwards analysis for predicting the minimum acceptance levels for EQ (sub)characteristics that ensure the required level of QiU  We have shown how to build a BNs for conducting such analysis Future work  Empirically validate the proposal by:  exercising the BNs in several contexts of use.  adapting the BNs to the specific peculiarities of each context  checking that the BNs learn as they should for these contexts  Refine this approach by combining it with some Principal Component Analysis during the initial definition of the BN  Provide a useful input to the ISO Working Group defining the new SQUARE family of standards based on our researches