1. Validation of the Method Adoption Model for Functional Size Measurement of Web Applications Silvia Abrahão Valencia University of Technology, Spain sabrahao@dsic.upv.es Geert Poels Ghent University, Belgium geert.poels@UGent.be 2nd Workshop on Software Metrics Validation Ghent, July 06, 2004

2. QAOOSE 2004 Contents Functional Size Measurement (FSM)  How to evaluate FSM methods? A Process Model for FSM A Theoretical Model for Evaluating FSM Methods A Laboratory Experiment to test the Theoretical Model Summary of Findings Next Steps

3. Ghent, July 06, 2004 QAOOSE 2004 Functional Size Measurement Functional size measurement (FSM) methods  Define functional size measures  Describe how to apply functional size measurement New FSM methods for new technologies  E.g. OO-Method Function Points (OOmFP) and its extension to the Web (OOmFPWeb) No methodology for systematic evaluation of FSM methods Little evidence of validity of functional size measures

4. Ghent, July 06, 2004 QAOOSE 2004 How to evaluate FSM methods? OOmFPWeb: A FSM method for Web Applications OOmFPWeb Measurement Procedure

5. Ghent, July 06, 2004 QAOOSE 2004 Possibly existing solutions Theoretical validation  Verification of empirical and numerical properties using Measurement Theory  Conformity evaluation to ISO/IEC 14143-1:1998 International Standard for FSM concepts Empirical validation  Demonstrate the relationship with relevant software process and management variables  Evaluate performance properties of FSM method (e.g. using ISO/IEC TR 14143-3:2003)  Pragmatic approach: method’s success in practice

6. Ghent, July 06, 2004 QAOOSE 2004 Our position Comprehensive evaluation framework needed  Multi-perspective evaluation  Systematic evaluation Based on software measurement process model  Design => Application => Analysis => Use Based on multi-disciplinary approach  Measurement Theory  IT / IS method evaluation models  Empirical software engineering

7. Ghent, July 06, 2004 QAOOSE 2004 A Process Model for FSM Validation of the design of the FSM method Validation of the application of the FSM method Validation of the models that use the FSM method results Includes measure validation Includes evaluation of performance-based and perception-based properties of the method Includes evaluation of model properties (e.g. prediction accuracy, causality) Measurement Theoretic approach Evaluation model for FSM methods Empirical software engineering approach Application of the measurement method rules Design of the Measurement method Measurement result Explotation of the measurement method result Step 1 Step 2Step 3 Step 4 (Jacquet and Abran, 1997)

8. Ghent, July 06, 2004 QAOOSE 2004 Method should enable task to be performed faster, more cheaply or with less effort (efficiency) and/or improve the quality of the result (effectiveness)  Efficiency is defined by the effort required to apply the method.  Effectiveness is defined by how well the method achieves its objectives.  Efficacy is the combination of efficiency and effectiveness. (Moody, 2001) The Method Evaluation Model An Overview Task Inputs Outputs Method reduce inputs (increase efficiency) improve outputs (increase effectiveness)

9. Ghent, July 06, 2004 QAOOSE 2004 Validation of the application of OOmFPWeb Method Evaluation Model (MEM): a theoretical model for evaluating IS design methods, which incorporates both aspects of method “success”: efficacy and adoption in practice. (Moody, 2001)

10. Ghent, July 06, 2004 QAOOSE 2004 Theoretical Model for Evaluating FSM Methods Model ’ s variables:  Measurement Time: the time taken by a subject to size a OOWS conceptual schema.  Reproducibility: the agreement between the measurement results of different subjects using OOmFPWeb.  Perceived Ease of Use: the degree to which a subject believes that using OOmFPWeb would be free of effort.  Perceived Usefulness: the degree to which a subject believes that OOmFPWeb will be effective in achieving its intended objectives.  Intention to Use: the degree to which an individual intends to use OOmFPWeb as a result of her perception of the method’s efficacy.

11. Ghent, July 06, 2004 QAOOSE 2004 Laboratory experiment to test the Theoretical Model  Analyze functional size measurements  For the purpose of evaluating OOmFPWeb  With respect to its efficacy and likely adoption in practice  From the point of view of the researcher.  The context of the experiment was an OOWS conceptual schema of a Web application that is measured by PhD students in the Department of Computer Science at the Valencia University of Technology. Validation of the application of OOmFPWeb

12. Ghent, July 06, 2004 QAOOSE 2004 Research Questions The research questions addressed were:  RQ1: Is OOmFPWeb efficacious?  RQ2: Is OOmFPWeb likely to be adopted in practice?  RQ3: Is Method Adoption Model a valid theoretical model for evaluating OOmFPWeb?

13. Ghent, July 06, 2004 QAOOSE 2004 Hypothesis Formulation We therefore test the following hypotheses:  Hypothesis 1: OOmFPWeb is efficient when compared to current industry practices.  Hypothesis 2: OOmFPWeb is effective when compared to similar studies reported in the literature.  Hypothesis 3: OOmFPWeb is perceived as easy to use.  Hypothesis 4: OOmFPWeb is perceived as useful.  Hypothesis 5: There is an intention to use OOmFPWeb.

14. Ghent, July 06, 2004 QAOOSE 2004 H1:Efficiency: Productivity of subjects X Reported industry averages The mean measurement productivity that was observed (108.79 FP/hour) is about three times the size of the industry benchmark. H2:Effectiveness: Reproducibility X Previous studies Compared to the results reported by Kemerer the consistency of measurements was high (mean REPi was 6%). The variation around the mean subject assessment (i.e. range of values divided by mean value) was 24.5%, comparing well with Rudolph’s study. Results Obtained

15. Ghent, July 06, 2004 QAOOSE 2004 H3, H4 and H5: PEOU, PU and ITU : survey instrument included 14 closed questions Example of a question to measure PEOU: These hypotheses was tested by verifying whether the scores that students assign to the constructs of the MAM are significantly better than the middle score (i.e. the score 3). All hypothesis were confirmed Results Obtained

16. Ghent, July 06, 2004 QAOOSE 2004 Analyzing the Validity and Reliability of the MAM Constructs Validity Analysis: inter-item correlation analysis was carried out. Reliability Analysis: inter-rater reliabilities .7 considered to be acceptable. CONSTRUCT CRONBACH ’ S α Perceived Ease of Use0,70 Perceived Usefulness0,75 Intention to Use0,80

17. Ghent, July 06, 2004 QAOOSE 2004 Regression Models

18. Ghent, July 06, 2004 QAOOSE 2004 Research QuestionResult RQ1: Efficiency 1.1 OOmFPWeb is efficient when compared to current industry practices Yes RQ1: Effectiveness 1.2 OOmFP is effective when compared to similar studies reported in the literature. Yes RQ2: Perception of Efficacy and Intention to Use 2.1 OOmFPWeb is perceived as easy to useYes 2.2 OOmFPWeb is perceived as usefulYes 2.3 There is an intention to use OOmFPWebYes RQ3: Validation of the Method Adoption Model 3.1 Perceived ease of use is determined by productivityNo 3.2 Perceived usefulness is determined by reproducibilitySi 3.3 Perceived usefulness is determined by perceived ease of useNo 3.4 Intention to use is determined by perceived ease of use and perceived usefulness Partially. Summary of Findings

19. Ghent, July 06, 2004 QAOOSE 2004 Work done so far.. Developing and validating measurement instruments for perception-based variables Three replications of the study presented here: 1.Valencia University of Technology, Spain  46 students in the last year of Computer Science degree 2.Valencia University of Technology, Spain  18 PhD students 3.University of Klagenfurt, Austria.  28 Master’s students

20. Ghent, July 06, 2004 QAOOSE 2004 Next Steps Analyzing the collected data to verify whether the proposed theoretical model is a valid model and measurement instrument for evaluating FSM methods. A new experiment using practitioners of OOWS in the context of the Spanish Association of Software Metrics.