1. Quality Determination
for Web-based Applications
Hazura Zulzalil

2. Outlines Overview of MCDA Evaluation of WBA General definition
MCDM process
MCDA methods
Evaluation of WBA
Quality Attribute Relationships
Aggregation by Choquet Integral
Implementation
Case study and results

3. What is MCDA?
Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.

4. What is MCDM?
Highly structured, disciplined and formal approach to decision making
evaluating the alternatives in the given set A against the set C of criteria
Aggregating the individual evaluations to produce global evaluation
Could be used for selection the best possible alternatives or for ranking the alternatives

5. MCDM Process Set of Alternatives Set of Criteria Weights wi /
Importance of Criteria
Overall worth of an alternative Ai
Aggregation Measure
C1, C2,………Cn
A1 x11……..………x1n
A2 x21……..………x2n .
Am xn1……..………xmn

6. Evaluation of MCDA methods
Criteria – interdependence, completeness, non-linear preferences
Weights – transparency of process, type of weights, meaning
Solution finding procedure – ranking, option
Project constraints – cost, time

7. Evaluation of MCDA methods
Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication
Data Situation
Type of data - qualitative or quantitative
Risk/uncertainties – probabilities, thresholds, fuzzy numbers, sensitive analysis
Data processing amount
Non-substitutability

8. Evaluation of WBA Quality of web site is hard to evaluate
Consists of multiple criteria to be measured
Simple weighted average cannot be used to summaries the various quality measurements into a single score.
Inability to account for dependency among the quality criterion.
Tend to construct independent criteria, or criteria that are supposed to be so
Causing some bias effect in evaluation

9. WBA Evaluation Approaches
Single criteria
usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005),
content and structure (Bauer & Scharl, 2000).
accessibility (Vigo et al., 2007)

10. WBA Evaluation Approaches
Multi-criteria
WEBQEM (Olsina et al., 1999)
EWAM (Schubert & Selz, 1998)
WebQual (Barnes and Vidgen, 2002)
WAI (Miranda et al., 2006)
FQT4Web (Davoli et al., 2005)

11. ISO/IEC 9126 Evaluation Process
Stated or implied needs
Quality
Requirement
Definition
Software Development
ISO 9126 & other technical info
Quality requirement specification
Metric Selection
Rating level definition
Assessment criteria definition
Measurement
Rating
Assessment
Products
Measured value
Rated value
Result (acceptable or unacceptable)
Requirement definition
Managerial requirement
Preparation
Evaluation

12. Quality Model Indicators, scales and preferred values APPLICATION
DOMAIN
e-commerce
e-learning
e-education
e-government
etc. Q U A L I T Y C H R E S
Functionality
Reliability
Usability
Efficiency
Portability
Maintainability
suitability
accuracy
interoperability
security
traceability
functionality
compliance
maturity
fault tolerance
recoverability
availability
degradability
reliability
understandability
learnability
operability
attractiveness
expliciteness
customisability
clarity
helpfulness
user-friendliness
usability compliance
time behaviour
resource
utilisation
efficiency
Analysability
changeability
stability
testability
manageability
reusability
maintainability
Adaptability
installability
coexistence
replaceability
portability S U B C H A R T E I
Indicators, scales and preferred values

13. Quality Attributes for WBA
Define software product qualities as a hierarchy of factors, criteria and metrics.
Quality factor represents behavioral characteristics of the system
Quality criterion is an attribute of a quality factor that is related to software production and design
Quality metrics is a measure that captures some aspect of a quality criterion.

14. Factor A is split up into three criteria a1, a2, and a3
Factor A is split up into three criteria a1, a2, and a3. Criteria a1 with the weight 4 is considered four times as important as criteria a2 and twice as important as criteria a3.
Similarly, we can set different weight for each factor to indicate its importance.
Overall Quality Score
Factor A
Factor B
Factor C
Criteria a1, weight 4
Criteria a2, weight 1
Criteria a3, weight 2

15. Definition of Quality Attributes
Name
Description
Functionality
The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions
Usability
The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions
Reliability
The capability of the Web site to maintain a specified level of performance when used under specified conditions.
Efficiency
The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions
Maintainability
The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications
Portability
The capability of the site to be transferred from one environment to another

16. Quality Attributes Relationships
Three types of relationships
Positive, i.e. a good value of one attribute result in a good value of the other (synergistic goals).
Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be enhanced (+)
Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals).
Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be degraded (-)
Independent, i.e. the attributes do not affect each other.
Relationships definitions: If characteristics A is enhanced, then characteristics B is unlikely to be affected (0)

17. Interrelationships between quality factors (Perry, 1987)

18. Relationship Chart (Gillies, 1997)

19. Techniques to explore the relationships
Ref
Attributes
Purpose
Techniques used  
[8, 9]
Correctness, Reliability
Integrity, Usability
Efficiency, Maintainability
Testability, Flexibility
Portability. Reusability
Interoperability
To study the relations of different quality goals attribute in developing software
Survey -questionnaire
[10]
Performance
Adaptability
Maintainability
To address the importance of design decision made during software development
Case Study - Interview
[11]
Usability
Time to market
Reliability, Usability
Correctness, Portability
To increase the understanding of software quality attributes and their relations
Research Literature and Survey –structured interview
[12]
Quality attributes in 3 different perspectives: management, developer and user perspective
To merge different view and discuss the relationships between the quality attributes
Discussion (meeting and offline discussion)

20. Quality Attributes Relationships for WBA

21. What is Aggregation?
method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values

22. Aggregation issues use simple weighted average approach
methods are not transparent
assume independency
the choice of summarization method somehow should depend on the certain parameters
E.g. the kind of importance parameters (weights) and the type of dependency and interaction
the definition of the quality factors and their relationships must be clearly specified

23. Common aggregation operators
Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.)
Not stable under linear transformation and consider criteria as non interacting
Median
Typical ordinal operator – defined the middle value of the ordered list
Weighted minimum and maximum
Possible to increase one of the weights without having any change in the result
Ordered weighted averaging operators
Can express vague quantifiers
The easiest way to aggregate is the simple arithmetic mean. Many other mean exist, such as geometric, harmonic means, quadratic mean, root power means. 23 23

24. Properties of an aggregation operator
mathematical properties
Properties of extreme values
Idempotence
Continuity
Monotonicity
Commutativity
Decomposability
Stability under the same positive linear transformation

25. Properties of an aggregation operator
behavioural properties
express the decisional behavior, interaction between criteria, interpretability of the parameters and weights on the arguments

26. Aggregation by fuzzy integral
Different methods have been developed according to
type of information to be aggregated and
the properties have to be satisfied. 26

27. Fuzzy measures and integral
Definition 1: A fuzzy measure on the set X of criteria is a set function  : Ƥ (X) [0,1], satisfying the following axioms
 ()=0,  (X)=1.
A  B  X implies (A)  (B)
(A) represent the weight of importance of the set of criteria A.
Additive : if (AB) = (A) + (B); A  B=
Superadditive: if (AB)  (A) + (B); A  B=
Subadditive if (AB)  (A) + (B); A  B=
If a fuzzy measure is additive, then it suffices to define n coefficients (weights) ({ I}), … ({ n}) 27

28. Choquet integral
Definition 2: Let  be a fuzzy measure on X. The choquet integral of a function ƒ : (X) [0,1] with respect to  is defined by
C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i) )
ƒ ((0)) = 0 n
i = 1
Fuzzy integral model does not need to assume independency
Fuzzy integral of ƒ with respect to  gives the overall evaluation of an alternative 28

29. Importance and interaction of criteria
Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L).
By weighted sum (3 , 3, 2) result: 29

30. Solved by fuzzy measure  and the choquet integral
Scientific subjects are more important than literature;
 ({M}) =  ({P}) =0.45;  ({L}) = 0.3
M and P are redundant,
 ({M, P}) = 0.5 <
Students equally good at scientific subjects and literature,
 ({L, M}) = 0.9 >
 ({L, P}) = 0.9 >
 ()=0,  ({M, P, L})=1 30

31. Result by applying fuzzy measure:
Result by applying fuzzy measure: * The initial ratio of weight (3, 3, 2) is kept unchanged 31

32. Complexity of the model
Number of coefficients grows exponentially with the number of criteria to be aggregated.
3 approaches (to reduce the number of coefficients)
Identification based on semantics
Importance of criteria
Interaction between criteria
Symmetric criteria
Veto effects
Identification based learning data
Minimization of squared error
Constraint satisfaction
Combining semantics and learning 32

33. Proposed solution Apply 2-additive Choquet integral
provide the information about the relationships among criteria (redundancy or support among criteria) and the preference among alternatives
Derive fuzzy measures by constraint satisfaction

34. Explore relationships
Techniques to explore how the different attributes are related to each other:
Experience Based Approach
Mathematical Modeling
Statistical Technique (Correlation Analysis)
measures the strength of a linear relationship among different quality factors
The main result of a correlation is called the correlation coefficient (r)

35. Correlation Result

38. Implementation of Choquet Integral
Definition of the initial preferences.
Convert into Choquet integral form
Identify threshold values.
If solution exists, calculate the Choquet integral, Shapley value and Interaction indices

39. Define preference
thresholds

40. Convert into Choquet integral form

41. Define preference thresholds
Three preference thresholds C, Sh & I have to be determined before the aggregation take part.
Range of : 0 to 1
no rule to fix the , we need to compare the solutions obtain with different value of .
Once the solution exist, Choquet integral will be calculated

42. Calculate the Choquet integral

43. Calculate the Shapley value
with
Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions.
Summation of these Shapley values for a given set of elements would represent the importance of the complete set

44. Calculate the Interaction Index
With
The interaction index Iij can be interpreted as a kind of average value of the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is said to be positive (resp. negative).

45. Case Study Perform on 3 types of WBA
Academic
E-commerce
Museum
Four quality factor were evaluated
Usability,Functionality, Reliability, Efficiency
Each has different preference, importance and interaction

46. Result for academic website

48. Threshold
C= 1,
Sh = 0.1,
I =0.1,

51. Summary(1)

52. Summary(2)

53. Comparison with other approaches

54. Conclusion
Aggregation by Choquet integral can be alternated if there is interaction exist between quality factors.
The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures.
Results show that the global evaluation obtained is compatible with the weighted average method.

55. Future works
The evaluation of WBA which cater the dynamic changes of the quality factors.
Behavior (Preferences, importance,interaction, etc.) can be change continuously.
Investigate more than 2 quality attribute interactions