Quality Determination for Web-based Applications Hazura Zulzalil

Outlines Overview of MCDA Evaluation of WBA General definition MCDM process MCDA methods Evaluation of WBA Quality Attribute Relationships Aggregation by Choquet Integral Implementation Case study and results

What is MCDA? Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.

What is MCDM? Highly structured, disciplined and formal approach to decision making evaluating the alternatives in the given set A against the set C of criteria Aggregating the individual evaluations to produce global evaluation Could be used for selection the best possible alternatives or for ranking the alternatives

MCDM Process Set of Alternatives Set of Criteria Weights wi / Importance of Criteria Overall worth of an alternative Ai Aggregation Measure C1, C2,………Cn A1 x11……..………x1n A2 x21……..………x2n . Am xn1……..………xmn

Evaluation of MCDA methods Criteria – interdependence, completeness, non-linear preferences Weights – transparency of process, type of weights, meaning Solution finding procedure – ranking, option Project constraints – cost, time

Evaluation of MCDA methods Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication Data Situation Type of data - qualitative or quantitative Risk/uncertainties – probabilities, thresholds, fuzzy numbers, sensitive analysis Data processing amount Non-substitutability

Evaluation of WBA Quality of web site is hard to evaluate Consists of multiple criteria to be measured Simple weighted average cannot be used to summaries the various quality measurements into a single score. Inability to account for dependency among the quality criterion. Tend to construct independent criteria, or criteria that are supposed to be so Causing some bias effect in evaluation

WBA Evaluation Approaches Single criteria usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005), content and structure (Bauer & Scharl, 2000). accessibility (Vigo et al., 2007)

WBA Evaluation Approaches Multi-criteria WEBQEM (Olsina et al., 1999) EWAM (Schubert & Selz, 1998) WebQual (Barnes and Vidgen, 2002) WAI (Miranda et al., 2006) FQT4Web (Davoli et al., 2005)

ISO/IEC 9126 Evaluation Process Stated or implied needs Quality Requirement Definition Software Development ISO 9126 & other technical info Quality requirement specification Metric Selection Rating level definition Assessment criteria definition Measurement Rating Assessment Products Measured value Rated value Result (acceptable or unacceptable) Requirement definition Managerial requirement Preparation Evaluation

Quality Model Indicators, scales and preferred values APPLICATION DOMAIN e-commerce e-learning e-education e-government etc. Q U A L I T Y C H R E S Functionality Reliability Usability Efficiency Portability Maintainability suitability accuracy interoperability security traceability functionality compliance maturity fault tolerance recoverability availability degradability reliability understandability learnability operability attractiveness expliciteness customisability clarity helpfulness user-friendliness usability compliance time behaviour resource utilisation efficiency Analysability changeability stability testability manageability reusability maintainability Adaptability installability coexistence replaceability portability S U B C H A R T E I Indicators, scales and preferred values

Quality Attributes for WBA Define software product qualities as a hierarchy of factors, criteria and metrics. Quality factor represents behavioral characteristics of the system Quality criterion is an attribute of a quality factor that is related to software production and design Quality metrics is a measure that captures some aspect of a quality criterion.

Factor A is split up into three criteria a1, a2, and a3 Factor A is split up into three criteria a1, a2, and a3. Criteria a1 with the weight 4 is considered four times as important as criteria a2 and twice as important as criteria a3. Similarly, we can set different weight for each factor to indicate its importance. Overall Quality Score     Factor A Factor B Factor C Criteria a1, weight 4 Criteria a2, weight 1 Criteria a3, weight 2

Definition of Quality Attributes Name Description Functionality   The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions Usability   The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions Reliability   The capability of the Web site to maintain a specified level of performance when used under specified conditions. Efficiency   The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions Maintainability The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications Portability   The capability of the site to be transferred from one environment to another

Quality Attributes Relationships Three types of relationships Positive, i.e. a good value of one attribute result in a good value of the other (synergistic goals). Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be enhanced (+) Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals). Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be degraded (-) Independent, i.e. the attributes do not affect each other. Relationships definitions: If characteristics A is enhanced, then characteristics B is unlikely to be affected (0)

Interrelationships between quality factors (Perry, 1987)

Relationship Chart (Gillies, 1997)

Techniques to explore the relationships Ref Attributes Purpose Techniques used   [8, 9] Correctness, Reliability Integrity, Usability Efficiency, Maintainability Testability, Flexibility Portability. Reusability Interoperability To study the relations of different quality goals attribute in developing software Survey -questionnaire [10] Performance Adaptability Maintainability To address the importance of design decision made during software development Case Study - Interview [11] Usability Time to market Reliability, Usability Correctness, Portability To increase the understanding of software quality attributes and their relations Research Literature and Survey –structured interview [12] Quality attributes in 3 different perspectives: management, developer and user perspective To merge different view and discuss the relationships between the quality attributes Discussion (meeting and offline discussion)

Quality Attributes Relationships for WBA

What is Aggregation? method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values

Aggregation issues use simple weighted average approach methods are not transparent assume independency the choice of summarization method somehow should depend on the certain parameters E.g. the kind of importance parameters (weights) and the type of dependency and interaction the definition of the quality factors and their relationships must be clearly specified

Common aggregation operators Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.) Not stable under linear transformation and consider criteria as non interacting Median Typical ordinal operator – defined the middle value of the ordered list Weighted minimum and maximum Possible to increase one of the weights without having any change in the result Ordered weighted averaging operators Can express vague quantifiers The easiest way to aggregate is the simple arithmetic mean. Many other mean exist, such as geometric, harmonic means, quadratic mean, root power means. 23 23

Properties of an aggregation operator mathematical properties Properties of extreme values Idempotence Continuity Monotonicity Commutativity Decomposability Stability under the same positive linear transformation

Properties of an aggregation operator behavioural properties express the decisional behavior, interaction between criteria, interpretability of the parameters and weights on the arguments

Aggregation by fuzzy integral Different methods have been developed according to type of information to be aggregated and the properties have to be satisfied. 26

Fuzzy measures and integral Definition 1: A fuzzy measure on the set X of criteria is a set function  : Ƥ (X) [0,1], satisfying the following axioms  ()=0,  (X)=1. A  B  X implies (A)  (B) (A) represent the weight of importance of the set of criteria A. Additive : if (AB) = (A) + (B); A  B= Superadditive: if (AB)  (A) + (B); A  B= Subadditive if (AB)  (A) + (B); A  B= If a fuzzy measure is additive, then it suffices to define n coefficients (weights) ({ I}), … ({ n}) 27

Choquet integral Definition 2: Let  be a fuzzy measure on X. The choquet integral of a function ƒ : (X) [0,1] with respect to  is defined by C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i) ) ƒ ((0)) = 0 n i = 1 Fuzzy integral model does not need to assume independency Fuzzy integral of ƒ with respect to  gives the overall evaluation of an alternative 28

Importance and interaction of criteria Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L). By weighted sum (3 , 3, 2) result: 29

Solved by fuzzy measure  and the choquet integral Scientific subjects are more important than literature;  ({M}) =  ({P}) =0.45;  ({L}) = 0.3 M and P are redundant,  ({M, P}) = 0.5 < 0.45 + 0.45 Students equally good at scientific subjects and literature,  ({L, M}) = 0.9 > 0.45 + 0.3  ({L, P}) = 0.9 > 0.45 + 0.3  ()=0,  ({M, P, L})=1 30

Result by applying fuzzy measure: Result by applying fuzzy measure: * The initial ratio of weight (3, 3, 2) is kept unchanged 31

Complexity of the model Number of coefficients grows exponentially with the number of criteria to be aggregated. 3 approaches (to reduce the number of coefficients) Identification based on semantics Importance of criteria Interaction between criteria Symmetric criteria Veto effects Identification based learning data Minimization of squared error Constraint satisfaction Combining semantics and learning 32

Proposed solution Apply 2-additive Choquet integral provide the information about the relationships among criteria (redundancy or support among criteria) and the preference among alternatives Derive fuzzy measures by constraint satisfaction

Explore relationships Techniques to explore how the different attributes are related to each other: Experience Based Approach Mathematical Modeling Statistical Technique (Correlation Analysis) measures the strength of a linear relationship among different quality factors The main result of a correlation is called the correlation coefficient (r)

Correlation Result

Implementation of Choquet Integral Definition of the initial preferences. Convert into Choquet integral form Identify threshold values. If solution exists, calculate the Choquet integral, Shapley value and Interaction indices

Define preference thresholds

Convert into Choquet integral form

Define preference thresholds Three preference thresholds C, Sh & I have to be determined before the aggregation take part. Range of : 0 to 1 no rule to fix the , we need to compare the solutions obtain with different value of . Once the solution exist, Choquet integral will be calculated

Calculate the Choquet integral

Calculate the Shapley value with Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions. Summation of these Shapley values for a given set of elements would represent the importance of the complete set

Calculate the Interaction Index With The interaction index Iij can be interpreted as a kind of average value of the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is said to be positive (resp. negative).

Case Study Perform on 3 types of WBA Academic E-commerce Museum Four quality factor were evaluated Usability,Functionality, Reliability, Efficiency Each has different preference, importance and interaction

Result for academic website

Threshold C= 1, Sh = 0.1, I =0.1,

Summary(1)

Summary(2)

Comparison with other approaches

Conclusion Aggregation by Choquet integral can be alternated if there is interaction exist between quality factors. The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures. Results show that the global evaluation obtained is compatible with the weighted average method.

Future works The evaluation of WBA which cater the dynamic changes of the quality factors. Behavior (Preferences, importance,interaction, etc.) can be change continuously. Investigate more than 2 quality attribute interactions