1. IT Research Seminar February 10, 2003
Measuring IS Success: Quest for the Dependent Variable in IS Research The Journey Special IT Research Seminar William H. DeLone Kogod School of Business February 8, 2008
IT Research Seminar
February 10, 2003

2. IS Success Research Stream
UCLA Dissertation – Successful use of IS by SMEs; MISQ 1988
IS Success: Quest for the Dependent Variable; ISR 1992 = “DeLone & McLean Success Model”
Ten Year Update; JMIS, Spring 2003
Measuring E-Commerce Success, International Journal of Electronic Commerce, Fall 2004
IS Success Models, Dimensions, Measures, and Interrelationships, under review European Journal of Information Systems

3. Research Motivation UCLA measurement course (Mason & Swanson)
Peter Keen’s 5 research challenges for MIS (ICIS 1) – What is the dependent variable? How does MIS establish a cumulative tradition?
Dependent variable for PhD study in SMEs; Use & Impact
If you can’t measure it, you can’t research it

4. Quest for MIS Dependent Variable (ISR, 1992) – “IS Success”
Purposes
Organize & summarize MIS research related to defining the dependent variable
Measure progress on defining the dependent variable
Improve IS research practice
Contribute to “Cumulative Tradition” – compare apples to apples

5. Theoretical Underpinnings
Mason’s 1978 article on measuring information output
Production->Product->Receipt->Influence on Recipient->Influence on System
Mason’s work was based on Shannon & Weaver’s 1949 Theory of Communications book – Levels of communications measurement = technical, semantic, effectiveness
DeLone & McLean Success Categories
System Quality->Information Quality-> Use->Satisfaction->Individual Impact-> Organization Impact

6. Methodolgy Literature review IS Articles from 1981 to 1988
Framework/model for organizing success measures
Empirical measures grouped into six success categories

7. D & M IS Success Model User Use Individual Impact System Quality
Satisfaction
Use
Individual
Impact
Organizational
System
Quality
Information
Figure 1
D&M IS Success Model

8. Results/Conclusions
A simple and parsimonious framework for organizing IS success measures
IS Success - multi-dimensional and interdependent construct
Selection of measures is contingent on objectives and context of study
Reduce IS Success measures; build on existing measures=> “cumulative tradition” – comparison of results
Need for organizational impact measures

9. The Challenge
“ This success model clearly needs further development and validation before it could serve as a basis for the selection of appropriate IS measures.” (p.88)

10. Ten-Year Update (JMIS 2003)
Purposes
Model Utility
Validate the Model – Causal relationships
Update the Model to recognize the changes in IS
Assess progress in IS success measurement

11. Utility of D&M Success Model
Cited by more than 285 refereed journal and proceedings articles between 1993 and 2002 (according to a recent study in CAIS vol. 20, ISR 1992 article is the most cited article in MIS over the last 15 years; > 400 citations)
Most articles used the Model “as a drunkard uses a lamppost for support rather than illumination.” Statement was rejected by editor.

12. Model Validation
Seddon & Kiew (1994) validated 4 of the proposed associations
Rai et. al. validated overall model using goodness of fit tests (ISR 2002)
Fifteen additional empirical studies validated one or more proposed associations

13. Updated Model
IS move from production function to production & service function => importance of service quality
Impacts on whom? Individuals, groups, org., industry, economy => Net Benefits dimension with contextual definition

14. Updated Model

15. Assessment & Conclusions
D&M IS Success Model supported and validated
More careful attention to multidimensionality of IS Success
Confusion between Independent & Dependent variables
Operationalization of the model is contextual (Seddon et. al.)
Progress in parsimonious measure development is slow
System Use is misunderstood and undervalued
Use and satisfaction are not a substitute for Net Benefits measures (Yuthas & Young, 1998)

16. Recent Advances in Success Measurement

17. Application of Model: E-Commerce Success (IJEC 2004)
Premise: E-commerce does not need a new measurement paradigm but some new measures
Apply the D&M IS Success Model for measuring E-Commerce Success
Literature Review and classification of emerging e-commerce effectiveness measures
Case examples of application of IS Success Model to E-Commerce success measurement

18. Application of Model: ERP Success
Article by Sedera & Gable (ICIS 2004) – Government & University ERPs
Most comprehensive empirical test of model
Four dimensions of IS Success –
System quality
Information quality
Individual impact
Organizational impact
** 27 Item measures

19. Validated Measures for IS Success Source: Sedera and Gable (2004)
System Quality - ease of use, ease of learning, user requirements, system features, system accuracy, flexibility, sophistication, integration, and customization
Information Quality – availability, usability, understandability, relevance, format, and conciseness
Individual Impact – learning, awareness/recall, decision effectiveness, and individual productivity
Organizational Impact - organizational costs, staff requirements, cost reduction, overall productivity, improved outcomes/outputs, increased capacity, e-Government, and business process change
What happened to Use and User Satisfaction?

20. Measuring Systems/Information Usage (Don Marchand)
Don Marchand, Professor of Strategy & Information Management at IMD – Switzerland
20% of value realization is in deployment: 80% of value realization is in information and IT USAGE
Challenge – Information Usage is difficult to see, measure and manage

21. Measuring System Usage (Burton-Jones & Straub, ISR 2006)
Problem – over-simplified measures of use; e.g. duration of use and breadth of use
Importance of context & purpose
Elements of usage include: systems, user and task
Proposed Dimensions of Systems Usage – Cognitive Absorption (engagement) + Deep Structure (use of system features that support a specific task)
Systems Usage empirically related to task performance

22. Current Research
Measuring IS Success: Models, Dimensions, Measures and Interrelationships under review at European Journal of Information Systems
Assessing the state of IS Success Measurement (via D & M Model)
Summarizing empirical literature, 1992 to 2006
Contributions
Summarizes measures used for each dimension of success
Validates significant relationships for 10 of the 15 causal relationships in the Updated Success Model based on empirical studies

23. Conclusions
DeLone & McLean IS Success Model remains the most popular, comprehensive framework for guiding the development of the dependent variable in IS research and for comparing results
More IS researchers are using the D&M Model to inform and guide their measurement of the dependent variable rather than to merely justify their choice of measures
Information quality and information use are under studied
Satisfaction is over used as a surrogate for success; therefore much information is lost
Bias toward ease of data collection threatens rigor and understanding