Conceptual Model Evaluation. Towards more Epistemological Rigor Presentation at EMMSAD ’05 Jan Recker Queensland University of Technology June 14, 2005

Why Philosophy? [a] good part of the answer to the question “why philosophy?” is that the alternative to philosophy is not no philosophy but bad philosophy. The ‘unphilosophical’ person has an unconscious philosophy, which they apply in their practice – whether of science or politics or daily life. Collier (1994), p. 17 © Jan Recker, Queensland University of Technology

Agenda Motivation Philosophy in IS Research Conceptual Model Evaluation A Paradigmatic Discussion Framework Sample Application Conclusions © Jan Recker, Queensland University of Technology

The Need for Evaluation Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions “No problem-solving process can be considered complete until evaluation has been carried out. It is the evaluation which helps us to measure the effectiveness of the problem-solving process and the problem solver in the 'problem situation' – unless this element is considered there is no way of establishing that the 'problems' have been successfully resolved” Jayaratna (1994), p. 108 © Jan Recker, Queensland University of Technology

Scope, Idea, and Objective Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Scope: Evaluation of conceptual models Idea: Transfer Insights from Philosophy to the field of Evaluation: Paradigmatic Analysis Objective: Explicate the implications of paradigmatic presuppositions onto the evaluation of conceptual models Research method: Critique: Explicating and Discussing paradigmatic presuppositions onto the evaluation of conceptual models to determine possibilities, scope, and limits. © Jan Recker, Queensland University of Technology

Philosophy and IS Research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions IS discipline at the intersection of multiple research fields “State of diversity” Diverse approaches towards cognition, reality, truth etc. are being used © Jan Recker, Queensland University of Technology

Paradigms in IS Research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Paradigm: a specific way of thinking about problems based on a set of achievements that are acknowledged as a foundation of further research practice Avison & Fitzgerald (1995) Predominant paradigms in IS: Positivism Interpretivism Chen & Hirschheim (2004) © Jan Recker, Queensland University of Technology

Paradigmatic Framework Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Realism: things are out there, ergo they can be measured… © Jan Recker, Queensland University of Technology

Models in IS Research Conceptual Models Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Conceptual Models facilitate, systemize, and aid the process of information system engineering. describe object systems (e.g. an information system) of some domain in semantic terms, using an abstract yet formalized language. Core (if not the) artefact(s) of the discipline © Jan Recker, Queensland University of Technology

Evaluation in IS research Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Development of evaluation theory and practice is at the core of multiple disciplines Example: Stufflebeam (2001): twenty-two generic program evaluation approaches Evaluation methods differ substantially in perspective, understanding, and approach © Jan Recker, Queensland University of Technology

Selected Evaluation Approaches (empirical) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Survey This technique refers to the use of questionnaires to gather human attitudes, opinions, and impressions on methods. It mainly focuses on the perception of users. Laboratory Experiment This technique is used to assess the internal validity of modelling methods. In a laboratory experiment, independent variables, such as different modelling methods, are manipulated in order to measure the effect on dependent variables like accuracy or time costs of modelling. Field Experiment This technique is similar to laboratory experiments but the experiment takes place in natural settings. Researchers perform the investigation in concrete business organisations while trying to maintain control over the most significant independent variables. Case Study This technique focuses the systematic observation of a particular group or subject that utilises modelling methods. The investigation is conducted without intervening in the modelling process; instead focus is spent on gathering real life data on the usage of modelling methods. Action Research Action research is the application and testing of ideas developed in an academic environment in real world situations under participation of the researching individuals. By attending the modelling process, information about modelling experience through the participating researcher can be gathered. Cf. Siau & Rossi (1998) © Jan Recker, Queensland University of Technology

Selected Evaluation Approaches (non-empirical) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Feature Comparison This technique attempts to compare existing modelling methods by modelling the same domain with different methods and investigate how the various methods represent the same problem. The evaluation is then conducted based on a checklist of features that should ideally be included in the modelling methods. Metamodelling This technique uses meta models of methods as an analysis basis. It is attempted to evaluate methods by structurally investigating analogies and dissimilarities of their meta models. Metrics Approach This technique aims at comparing methods based on a predefined set of method metrics. Metric values are compared to reference values which address complexity and appropriateness of a method. Paradigmatic Analyses This technique refers to the analysis of underlying assumptions of methods, e. g., the view of IS development intention, the view on language functions, the definition of IS etc. Contingency Identification This technique aims at identifying the contingencies of the project in which a method is utilised. Therefore, criteria, such as the problem to be solved or the project team, are investigated to provide heuristics for the selection of an appropriate method. Ontological Evaluation This technique uses ontological concepts to evaluate modelling methods. The idea is to map existing modelling language constructs to the constructs provided by the ontology to assess the modelling language’s capability to represent reality. Cf. Siau & Rossi (1998) © Jan Recker, Queensland University of Technology

Preliminary Conclusions Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Plethora of model evaluation methods available Evaluation design is coined by philosophical presuppositions of designing and applying researchers Discussion of presuppositions aids explicating evaluation approaches Different world views of researchers designing evaluation approaches serve the basis for evaluation, e.g. by coining the understanding of model purpose and model quality © Jan Recker, Queensland University of Technology

Construction (1/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Investigation of existing model evaluation techniques based on Philosophical aspects Ontological aspect Epistemological aspect Paradigmatic approach upon aspects Positivist approach Interpretivist approach Paradigmatically coined perception of Models Evaluation Quality © Jan Recker, Queensland University of Technology

Construction (2/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions © Jan Recker, Queensland University of Technology

The BWW Ontology Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Wand and Weber applied an ontology based on Bunge’s work to the field of conceptual modeling The BWW Ontology specifies “reality” constructs that a conceptual modeling language should be able to depict The BWW ontology serves as a reference point in evaluation Evaluation is conducted by mapping language constructs against ontology constructs and thereby assessing ontological completeness and ontological deficiency within the modeling language Cf. Bunge (1977) Cf. Wand & Weber (1993) © Jan Recker, Queensland University of Technology

Applying the framework Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Representation Correspondence Clarity and Completeness © Jan Recker, Queensland University of Technology

Findings: Some Implications Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Application of BWW approach restricted to contexts adhering to same paradigm Positivist stance of BWW approach problematic “Positivism” should no longer even be mentioned in discussion of theory or epistemology in information systems as a defensible position. Gregor (2004), p. 4 Reconsideration and modificaiton of the BWW ontology? © Jan Recker, Queensland University of Technology

Conclusions Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Evaluation is problematic due to its dependancy on paradigmatic viewpoints onto Object of evaluation (models) Evaluation methodology (obtainment of truth) Evaluation target objective (quality) Multi-paradigm research during artefact design and evaluation is proscribed Need for more paradigmatic rigor in IS evaluation research a shift of paradigms during model development and model evaluation would resist reconciliation or synthesis. © Jan Recker, Queensland University of Technology

Thank you for listening Any Questions? © Jan Recker, Queensland University of Technology

Selected References (1/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Avison, D.E. and Fitzgerald, G. (1995) Information Systems Development: Methodologies, Techniques and Tools, McGraw-Hill Companies, London. Berger, P.L. and Luckmann, T. (1966) The Social Construction of Reality : A Treatise in the Sociology of Knowledge, Doubleday, Garden City. Bunge, M.A. (1977) Treatise on Basic Philosophy Volume 3: Ontology I - The Furniture of the World, Kluwer Academic Publishers, Dordrecht. Collier, A. (1994) Critical Realism: An Introduction to Roy Bhaskar's Philosophy, Verso, London. Gregor, S. (2004) "The Struggle Towards an Understanding of Theory in Information Systems" in D. Hart and S. Gregor (eds.) Information Systems Workshop: Constructing and Criticising, School of Business and Information Management, Canberra, pp. 1-11. Gruber, T.R. (1993) What is an Ontology?, Retrieved from http://www.ksl.stanford.edu/kst/what-is-an-ontology.html. © Jan Recker, Queensland University of Technology

Selected References (2/2) Motivation Philosophy in IS Model Evaluation Discussion Framework Application Conclusions Jayaratna, N. (1994) Understanding and Evaluating Methodologies: Nimsad, a Systematic Framework, McGraw Hill, New York et al. Losee, J. (2001) A Historical Introduction to the Philosophy of Science, Oxford University Press, Oxford. Siau, K. and Rossi, M. (1998) "Evaluation of Information Modeling Methods -- A Review" in D. Dolk (eds.) 31st Hawaii International Conference on System Sciences, Computer Society Press, Big Island, pp. 314-322. Stufflebeam, D.L. (2001) Evaluation Models. New Directions for Evaluation, Jossey-Bass, San Francisco. von Glasersfeld, E. (1987) The Construction of Knowledge. Contributions to Conceptual Semantics, Intersystems Publications, Seaside. Wand, Y. and Weber, R. (1993) On the Ontological Expressiveness of Information Systems Analysis and Design Grammars, Journal of Information Systems, 3, 217-237. © Jan Recker, Queensland University of Technology