1. Ghent, July 6, 2004 1 Evaluating a COSMIC-FFP Measurement Procedure for Multi-Layer Object-Oriented Conceptual Schemas Simon Claeys (Master student Ghent University) Geert Poels (Ghent University & Katholieke Universiteit Leuven)

2. 2 Ghent, July 6, 2004 Outline Research context  Development of a COSMIC-FFP measurement procedure for MERODE/OOWS Problem statement  How to evaluate this procedure? Possible solution  A laboratory experiment to test the procedure according to Abrahão’s general evaluation model for FSM methods

3. 3 Ghent, July 6, 2004 Research context – COSMIC-FFP “A more systematic and detailed procedure would provide precise mapping rules for a larger collection of highly specific artifacts, thus diminishing the level of ambiguity when generating the COSMIC-FFP generic software model. Such a procedure would, by definition, be highly dependent on the nature of the artifacts, which, in turn, depends on the software engineering methodology in use in each organization.” (COSMIC-FFP 2.2 Measurement Manual, p. 30)

4. 4 Ghent, July 6, 2004 Research context – Layered OO Conceptual Model BUSINESS DOMAIN MODEL FUNCTIONALITY MODEL PRESENTATION MODELNAVIGATION MODEL MERODE OOWS

5. 5 Ghent, July 6, 2004 Research context – Measurement procedure Mapping of concepts in the MERODE/OOWS meta-models onto the COSMIC-FFP meta-model COSMIC-FFP 2.2 mapping rules  4 rules for partial business domain model  4 rules for business domain model  7 rules for functionality model  5 rules for navigation model

6. 6 Ghent, July 6, 2004 Problem statement: How to evaluate this procedure ? March and Smith’s research framework for IT Research activities Design scienceNatural science BuildEvaluateTheorizeJustify Research outputs Constructs Model Method Instantiation "a set of steps (an algorithm or guideline) used to perform a task. Methods are based on a set of underlying constructs (language) and a representation (model) of the solution space.” (March and Smith 1995, p. 257)

7. 7 Ghent, July 6, 2004 Possible solution – Abrahão’s model The procedure (rules) should enable the task (applying COSMIC-FFP)  to be performed with less effort  and/or improve the quality of the result  compared to applying COSMIC-FFP without this procedure Task Inputs Outputs Method reduce inputs (increase efficiency) improve outputs (increase effectiveness)

8. 8 Ghent, July 6, 2004 Performance => Perceptions => Intentions => Behaviour Perceived Ease of Use Perceived Usefulness Intention to Use Actual Efficiency Actual Effective- ness PERFORMANCE PERCEPTIONS BEHAVIOUR Actual Usage INTENTIONS Actual Efficacy Perceived Efficacy Adoption in Practice

9. 9 Ghent, July 6, 2004 Operationalize performance properties using ISO/IEC TR 14143-3:2003  Repeatability and reproducability  Accuracy  Convertibility  Discrimination threshold  Applicability to functional domains A laboratory experiment is being planned  Spring 2005  Engineers enrolled in a post-graduate Master program in Industrial Management, specialising in ICT (at K.U.Leuven)  FSM with COSMIC-FFP as part of ICT Project Management course  Only a minority has studied MERODE

10. 10 Ghent, July 6, 2004 Possible solution – experimental design Test principle FUR Conceptual schema results COSMIC-FFP Proposed measurement procedure MERODE/OOWS COMPARE

11. 11 Ghent, July 6, 2004 Choices to be made:  Between-subjects versus within-subjects  Choice of application Reference model (ISO/IEC TR 14143-4:2002) Semantically correct MERODE case-study COSMIC-FFP case-studies  Form of the FUR (in the control group) Text ERD, DFD, UI prototype MERODE/OOWS schema  Data analysis and interpretation How to evaluate accuracy ?(calibration if systematic errors) How to measure reliability ? How relevant is measurement time ? How to interpret perception-based variables ?

12. 12 Ghent, July 6, 2004