1. Conceptual Model Evaluation. Towards more Epistemological Rigor
Presentation at EMMSAD ’05
Jan Recker
Queensland University of Technology
June 14, 2005

2. Why Philosophy?
[a] good part of the answer to the question “why philosophy?” is that the alternative to philosophy is not no philosophy but bad philosophy. The ‘unphilosophical’ person has an unconscious philosophy, which they apply in their practice – whether of science or politics or daily life.
Collier (1994), p. 17
© Jan Recker, Queensland University of Technology

3. Agenda Motivation Philosophy in IS Research
Conceptual Model Evaluation
A Paradigmatic Discussion Framework
Sample Application
Conclusions
© Jan Recker, Queensland University of Technology

4. The Need for Evaluation
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
“No problem-solving process can be considered complete until evaluation has been carried out. It is the evaluation which helps us to measure the effectiveness of the problem-solving process and the problem solver in the 'problem situation' – unless this element is considered there is no way of establishing that the 'problems' have been successfully resolved”
Jayaratna (1994), p. 108
© Jan Recker, Queensland University of Technology

5. Scope, Idea, and Objective
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Scope:
Evaluation of conceptual models
Idea:
Transfer Insights from Philosophy to the field of Evaluation: Paradigmatic Analysis
Objective:
Explicate the implications of paradigmatic presuppositions onto the evaluation of conceptual models
Research method:
Critique: Explicating and Discussing paradigmatic presuppositions onto the evaluation of conceptual models to determine possibilities, scope, and limits.
© Jan Recker, Queensland University of Technology

6. Philosophy and IS Research
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
IS discipline at the intersection of multiple research fields
“State of diversity”
Diverse approaches towards cognition, reality, truth etc. are being used
© Jan Recker, Queensland University of Technology

7. Paradigms in IS Research
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Paradigm:
a specific way of thinking about problems based on a set of achievements that are acknowledged as a foundation of further research practice
Avison & Fitzgerald (1995)
Predominant paradigms in IS:
Positivism
Interpretivism
Chen & Hirschheim (2004)
© Jan Recker, Queensland University of Technology

8. Paradigmatic Framework
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Realism: things are out there, ergo they can be measured…
© Jan Recker, Queensland University of Technology

9. Models in IS Research Conceptual Models
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Conceptual Models
facilitate, systemize, and aid the process of information system engineering.
describe object systems (e.g. an information system) of some domain in semantic terms, using an abstract yet formalized language.
Core (if not the) artefact(s) of the discipline
© Jan Recker, Queensland University of Technology

10. Evaluation in IS research
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Development of evaluation theory and practice is at the core of multiple disciplines
Example: Stufflebeam (2001):
twenty-two generic program evaluation approaches
Evaluation methods differ substantially in perspective, understanding, and approach
© Jan Recker, Queensland University of Technology

11. Selected Evaluation Approaches (empirical)
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Survey
This technique refers to the use of questionnaires to gather human attitudes, opinions, and impressions on methods. It mainly focuses on the perception of users.
Laboratory Experiment
This technique is used to assess the internal validity of modelling methods. In a laboratory experiment, independent variables, such as different modelling methods, are manipulated in order to measure the effect on dependent variables like accuracy or time costs of modelling.
Field Experiment
This technique is similar to laboratory experiments but the experiment takes place in natural settings. Researchers perform the investigation in concrete business organisations while trying to maintain control over the most significant independent variables.
Case Study
This technique focuses the systematic observation of a particular group or subject that utilises modelling methods. The investigation is conducted without intervening in the modelling process; instead focus is spent on gathering real life data on the usage of modelling methods.
Action Research
Action research is the application and testing of ideas developed in an academic environment in real world situations under participation of the researching individuals. By attending the modelling process, information about modelling experience through the participating researcher can be gathered.
Cf. Siau & Rossi (1998)
© Jan Recker, Queensland University of Technology

12. Selected Evaluation Approaches (non-empirical)
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Feature Comparison
This technique attempts to compare existing modelling methods by modelling the same domain with different methods and investigate how the various methods represent the same problem. The evaluation is then conducted based on a checklist of features that should ideally be included in the modelling methods.
Metamodelling
This technique uses meta models of methods as an analysis basis. It is attempted to evaluate methods by structurally investigating analogies and dissimilarities of their meta models.
Metrics Approach
This technique aims at comparing methods based on a predefined set of method metrics. Metric values are compared to reference values which address complexity and appropriateness of a method.
Paradigmatic Analyses
This technique refers to the analysis of underlying assumptions of methods, e. g., the view of IS development intention, the view on language functions, the definition of IS etc.
Contingency Identification
This technique aims at identifying the contingencies of the project in which a method is utilised. Therefore, criteria, such as the problem to be solved or the project team, are investigated to provide heuristics for the selection of an appropriate method.
Ontological Evaluation
This technique uses ontological concepts to evaluate modelling methods. The idea is to map existing modelling language constructs to the constructs provided by the ontology to assess the modelling language’s capability to represent reality.
Cf. Siau & Rossi (1998)
© Jan Recker, Queensland University of Technology

13. Preliminary Conclusions
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Plethora of model evaluation methods available
Evaluation design is coined by philosophical presuppositions of designing and applying researchers
Discussion of presuppositions aids explicating evaluation approaches
Different world views of researchers designing evaluation approaches serve the basis for evaluation, e.g. by coining the understanding of model purpose and model quality
© Jan Recker, Queensland University of Technology

14. Construction (1/2)
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Investigation of existing model evaluation techniques based on
Philosophical aspects
Ontological aspect
Epistemological aspect
Paradigmatic approach upon aspects
Positivist approach
Interpretivist approach
Paradigmatically coined perception of
Models
Evaluation
Quality
© Jan Recker, Queensland University of Technology

15. Construction (2/2) Motivation Philosophy in IS Model Evaluation
Discussion Framework
Application
Conclusions
© Jan Recker, Queensland University of Technology

16. The BWW Ontology
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Wand and Weber applied an ontology based on Bunge’s work to the field of conceptual modeling
The BWW Ontology specifies “reality” constructs that a conceptual modeling language should be able to depict
The BWW ontology serves as a reference point in evaluation
Evaluation is conducted by mapping language constructs against ontology constructs and thereby assessing ontological completeness and ontological deficiency within the modeling language
Cf. Bunge (1977)
Cf. Wand & Weber (1993)
© Jan Recker, Queensland University of Technology

17. Applying the framework
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Representation
Correspondence
Clarity and Completeness
© Jan Recker, Queensland University of Technology

18. Findings: Some Implications
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Application of BWW approach restricted to contexts adhering to same paradigm
Positivist stance of BWW approach problematic
“Positivism” should no longer even be mentioned in discussion of theory or epistemology in information systems as a defensible position.
Gregor (2004), p. 4
Reconsideration and modificaiton of the BWW ontology?
© Jan Recker, Queensland University of Technology

19. Conclusions
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Evaluation is problematic due to its dependancy on paradigmatic viewpoints onto
Object of evaluation (models)
Evaluation methodology (obtainment of truth)
Evaluation target objective (quality)
Multi-paradigm research during artefact design and evaluation is proscribed
Need for more paradigmatic rigor in IS evaluation research
a shift of paradigms during model development and model evaluation would resist reconciliation or synthesis.
© Jan Recker, Queensland University of Technology

20. Thank you for listening
Any Questions?
© Jan Recker, Queensland University of Technology

21. Selected References (1/2)
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Avison, D.E. and Fitzgerald, G. (1995) Information Systems Development: Methodologies, Techniques and Tools, McGraw-Hill Companies, London.
Berger, P.L. and Luckmann, T. (1966) The Social Construction of Reality : A Treatise in the Sociology of Knowledge, Doubleday, Garden City.
Bunge, M.A. (1977) Treatise on Basic Philosophy Volume 3: Ontology I - The Furniture of the World, Kluwer Academic Publishers, Dordrecht.
Collier, A. (1994) Critical Realism: An Introduction to Roy Bhaskar's Philosophy, Verso, London.
Gregor, S. (2004) "The Struggle Towards an Understanding of Theory in Information Systems" in D. Hart and S. Gregor (eds.) Information Systems Workshop: Constructing and Criticising, School of Business and Information Management, Canberra, pp
Gruber, T.R. (1993) What is an Ontology?, Retrieved from
© Jan Recker, Queensland University of Technology

22. Selected References (2/2)
Motivation
Philosophy in IS
Model Evaluation
Discussion Framework
Application
Conclusions
Jayaratna, N. (1994) Understanding and Evaluating Methodologies: Nimsad, a Systematic Framework, McGraw Hill, New York et al.
Losee, J. (2001) A Historical Introduction to the Philosophy of Science, Oxford University Press, Oxford.
Siau, K. and Rossi, M. (1998) "Evaluation of Information Modeling Methods -- A Review" in D. Dolk (eds.) 31st Hawaii International Conference on System Sciences, Computer Society Press, Big Island, pp
Stufflebeam, D.L. (2001) Evaluation Models. New Directions for Evaluation, Jossey-Bass, San Francisco.
von Glasersfeld, E. (1987) The Construction of Knowledge. Contributions to Conceptual Semantics, Intersystems Publications, Seaside.
Wand, Y. and Weber, R. (1993) On the Ontological Expressiveness of Information Systems Analysis and Design Grammars, Journal of Information Systems, 3,
© Jan Recker, Queensland University of Technology