1. Multilevel Model Analysis: Method and Applications
Eldon Y. Li
University Chair Professor
Department of MIS
National Chengchi University, Taiwan
*** All right reserved. Reference to this document should be made as follows: Li, E.Y. “Multilevel Model Analysis: Method and Applications”, unpublished lecture, National Chengchi University, 2018 ***
Copyright © E.Y.Li
2019/11/14

2. Agenda About me Abstract Introduction Why multilevel model?
When multilevel model?
How multilevel model?
Application example 1
Application example 2
Copyright (c) E.Y.Li
2019/11/14

3. About Me
Eldon Y. Li is a university chair professor and former department chair of MIS at National Chengchi University, an adjunct chair professor of Asia University in Taiwan, and a former professor and coordinator of the MIS program at College of Business, California Polytechnic State University, San Luis Obispo, California, USA. He was the dean of College of Informatics and the director of Graduate Institute of Social Informatics at Yuan Ze University in Taiwan, as well as professor and founding director at the Graduate Institute of Information Management at the National Chung Cheng University in Chia-Yi, Taiwan. He received his PhD from Texas Tech University in He is the editor-in-chief of several international journals. He has published more than 250 papers in various topics related to innovation and technology management, human factors in information technology (IT), strategic IT planning, software quality management, and information systems management. His papers appear in Journal of Management Information Systems, Research Policy, Communications of the ACM, Internet Research, Expert Systems with Applications, Computers & Education, Decision Support Systems, Information & Management, International Journal of Medical Informatics, Organization, among others.
2019/11/14

4. Multilevel Model Analysis: Method and Applications
Abstract
Conventional survey studies usually collect individual data from different groups and analyze them independently in each group, unless there is no significant group difference. In contrast, multilevel model analysis allows individual data with organizational differences to be included in one regression model by treating these differences as higher-level independent variables.  Such kind of model is known as hierarchical linear model (HLM). This lecture introduces various research models and discusses why, when, and how to perform multilevel model analysis. The applications of multilevel model analysis are elucidated using two published studies in information systems field.
Copyright (c) E.Y.Li
2019/11/14

5. Introduction – Model Analysis
Relational model
Causal model
Behavioural model
Process model
Mediation model
Moderation model
Moderated mediation model
Multilevel model
Mixed model

6. Relational model (Prediction model)
Source: Li, E.Y.* (1994) "Artificial Neural Networks and Their Business Applications," Information & Management (Elsevier), Vol. 27, No. 5, October, pp

7. Relational model (Prediction model)
Yen/Dollar Exchange (-1)
Nikkei 225
Yen/Dollar Exchange (-1.5)
Topix Composite
Yen/Dollar Exchange (-2)
Topix Large
Yen/Dollar Exchange (-3)
Topix Small
Source: Li, E.Y.* and Soenen, L. (1994) "Dollar Value of the Yen and Stock Price Reactions in Japan," Journal of Global Business (U.S.A.), Vol. 5, No. 1, Spring, pp

8. Causal model
Source: Wu, Y.L., Li, E.Y.*, and Chang, W.L. (2016) "Nurturing user creative performance in social media networks: an integration of habit of use with social capital and information exchange theories," Internet Research (Emerald), Vol. 26, No. 4, pp (SSCI)

9. Behavioral model Theory of Reasoned Action
Source: Fishbein, M., & Ajzen, I. (1975). Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley. 

10. Process model
Source: Bhattacherjee, A. and Premkumar, G. (2004). Understanding changes in belief and attitude toward information technology usage: a theoretical model and longitudinal test. MIS Quarterly, 28 (2),

11. Partial mediation model
Source: Huang, Y.H.*, Li, E.Y., and Chen, J.S. (2009.3) "Information Synergy As the Catalyst Between IT Capability and Innovativeness: Empirical Evidence from Financial Service Sector," Information Research: An International Electronic Journal, Vol. 14, No. 1, pp
Figure 1: Research Model

12. Full mediation model
Source: Yen, H.J.R., Li, E.Y. and Niehoff, B. (2008.9). Do organizational citizenship behaviors lead to information system success? testing the mediation effects of integration climate and project management. Information & Management, 45 (6),

13. Moderation model
Source: Venkatesh, V., Morris, M.G., Davis, G.B., and Davis, F.D. (2003). User Acceptance of information technology: Toward a unified view. MIS Quarterly, (27:3),

14. Moderated mediation model
Source: Ajzen, I. “TPB Diagram”, available at

15. Multilevel model
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp

16. Multilevel model
Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp

17. Mixed model
Source: DeLone, W. and McLean, E. (2003). The DeLone and McLean Model of information systems success: a ten-year update. Journal of Management Information Systems, 19 (4), 9-30.

18. Mixed model
Source: Wixom, B.H. and Todd, P.A. (2005). A theoretical integration of user satisfaction and technology acceptance. Information Systems Research, 16 (1),

19. Why multilevel model? Company 1 Company 2 Company 3 Company 4

20. When multilevel model? Condition for applying multilevel analysis
Variance between companies  relatively large
Variance within a company (between employees)  relatively small.
If variance between companies is small, use single level analysis.
If variance within a company (between employees) is large, remove this company from the data.

21. When? (Cont.) Intraclass agreement index (rwg)
Also called Within-organization agreement index
rwg (J) value indicates the degree to which the responses to a measurement scale by members of the same organization converge.
rwg (J) value >0.70 (James et al., 1984)
James LR, Demaree RG, Wolf G (1984) Estimating within-group interrater reliability with and without bias. J Appl Psychol 69:85response

22. When? (Cont.) Intraclass correlation coefficients
ICC1 compares the between-organizations variance with the within-organization variance to indicate the portion of variance in individual responses (MSW) accounted for by the between-organizations difference (MSB).
> 0.12 (Bliese 2000)
K = the average sample size from a company
Bliese, P.D. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K.J. Klein and S.W.J. Kozlowski (eds.), Multilevel Theory, Research, and Methods in Organizations. San Francisco: Jossey-Bass, 2000, pp. 349–381.

23. When? (Cont.) Intraclass correlation coefficients (cont.)
ICC2 reveals the reliability of the mean of an organization-level variable. If low reliability, multilevel analysis is needed.
That is, MSB should be large, MSW should be smaller and no more than than 40% of MSB.
> 0.60 (Bliese 2000)

24. How multilevel model?
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp

25. How? (Cont.)

26. Application example 1
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp
Copyright © E.Y.Li
2019/11/14

27. Application example 1 - Originality
The IS Success (ISS) model of DeLone and McLean (1992, 2003)
IS’s Qualities
User’s Satisfaction
Organization’s Net Benefits
IS Developers
Users
???
User Managers
Copyright © E.Y.Li
2019/11/14

28. Originality (Cont.)
The ISS model of DeLone and McLean (1992, 2003) prescribes IS’s quality (including information, system, and service qualities), user’s satisfaction, and organization’s net benefits as the three integrated components. While IS developer is responsible for IS quality, users are concerned with satisfaction, leaving net benefits unattended.

29. Research questions
What are the factors influencing net benfits of user department’s IS appcliation (as user manager’s job performance)?
What are the interaction effects of these factors on net benefits?

30. Underlying theories
Copyright © E.Y.Li
2019/11/14

31. Research model
Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp
Copyright © E.Y.Li
2019/11/14

32. Solution model

33. Solution model
UDISPij represents the ith individual score of UDISP in jth organization.
TMSj represents the aggregate score of TMS in jth organization.
ISKij represents the ith individual score of ISK in jth organization.
ISAij represents the ith individual score of ISA in jth organization.
γkl represents the slope of the kth level-1 predictor interacting with the lth level-2 predictor.
Ukj is a normal distribution and represents the residual of slope of kth level-1 predictor in the jth organization.
rij is a normal distribution and represents the residual of regression model in individual level.

34. HLM software for Windows

35. Model specification in HLM
Level Level-2
Coefficients Predictors
INTRCPT1, B0 INTRCPT2, G00
TMS, G01
ISK slope, B1 INTRCPT2, G10
TMS, G11
ISA slope, B2 INTRCPT2, G20
TMS, G21
Summary of the model specified (in equation format)
Level-1 Model
Y = B0 + B1*(ISK) + B2*(ISA) + R
Level-2 Model
B0 = G00 + G01*(TMS) + U0
B1 = G10 + G11*(TMS) + U1
B2 = G20 + G21*(TMS) + U2

36. Method - measures Job performance  User department’s IS performance.
Opportunity  Top management support.
Capability  User manager’s knowledge about IS applications.
Willingness  User manager’s attitude toward IS applications.
Legend:  =measured by.

37. Method - subjects Data requirements:
Using similar information systems: ERP
In similar industry: Manufacturing
System experience: At least 1 year
At least 10 companies in the industry: 42
At least 5 data ponits (departments) in each company: Average 6-7 deparments, Total 283

38. Analysis
Focus group was used to ensure face validity of the survey questionnaire.
Valid survey data were collected from 283 user managers and 42 top managers of 42 different Chinese manufacturing companies in which ERP systems were being utilized.
Each company sample=6 to 7  K=6; ICC1=0.253 >0.12; ICC2=0.670 >0.60.
The model can be validated by using Hierarchical Linear Modelling (HLM) software.

39. Results

40. Example 1 Conclusions
Top management support, user-manager knowledge, and user-manager attitude all affect the level of UDISP significantly. (H1, H2, H3 supported)
Top management support significantly moderates the relationship between user-manager attitude and UDISP. (H5 supported)
The interaction effect of top management support and user-manager knowledge on UDISP is not significant. (H4 not supported)

41. Application example 2
Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp

42. Solution model Department-level Model:
Employee’s Loyal Use: ELUij = β0j + β1j (EPBij ) +β2j (EPWij) +rij,
Company-level Model:
β0j = γ00 + γ01 (OLSQj) + γ02 (OLIQj) + γ03 (OLSOCBj) +U0j
β1j = γ10 + γ11 (OLSQj) + γ12 (OLIQj) + γ13 (OLSOCBj) +U1j
β2j = γ20 + γ21 (OLSQj) + γ22 (OLIQj) + γ23 (OLSOCBj) + U2j
The final formula is as follows:
ELUij = γ00 + γ01 (OLSQj) + γ02 (OLIQj) + γ03 (OLSOCBj) +U0j +(γ10 + γ11 (OLSQj) + γ12 (OLIQj) + γ13 (OLSOCBj) +U1j) EPBij +(γ20 + γ21 (OLSQj) + γ22 (OLIQj) + γ23 (OLSOCBj) + U2j)EPWij +rij,

43. Solution model
ELUij represents the ith individual score of ELU in jth organization.
OLSQj represents the aggregate score of OLSQ in jth organization.
OLIQj represents the aggregate score of OLIQ in jth organization.
OLSOCBj represents the aggregate score of OLSOCB in jth organization.
EPBij represents the ith individual score of EPB in jth organization.
EPWij represents the ith individual score of EPW in jth organization.
γkl represents the slope of the kth level-1 predictor interacting with the lth level-2 predictor.
Ukj is a normal distribution and represents the residual of slope of kth level-1 predictor in the jth organization.
rij is a normal distribution and represents the residual of regression model in individual level.

44. Underlying theories Individual Level Organization Level
Interactionism Paradigm
Situational Strength Theory
Employee’s
Perceived Workload
Social Information Processing Theory
Social Learning Theory
Rational Choice Theory
Cost–Benefit Analysis
Loyal Use
Individual Perceptions
Causal relation
Construct
Theoretical foundation
Bottom-up/Top-down process
Copyright © E.Y.Li
2019/11/14

45. IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02
Results
Individual Level
OL-IQ
OL-SOCB
Employee’s
Perceived Workload
Employee’s Loyal Use
Organization Level
OL-SQ H1
-0.11* ns
-0.12* H2
0.32***
0.47***
Perceived Benefits
IL-IQ
0.11*
-0.29***
IL-SQ
IL-SOCB
IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02
0.14*
-0.07***
0.23***

46. Results

47. Results

48. Results

49. Example 2 Conclusions
IL-SQ, IL-IQ, IL-SOCB all affects positively EPB.
Only IL-SQ affects negatively EPW.
IL-SQ and IL-IQ affects positively ELU.
EPB affects positively ELU.
EPW affects negatively ELU.
OL-SQ and OL-SOCB moderates negatively EPBELU.
OL-IQ moderates positively EPWELU.

50. Overall Conclusions
Multilevel model analysis overcomes the group differences and analyzes samples from multiple groups in one regression model.
When group differences are not significant, use single level analysis; when significant, use multilevel analysis.
Test rwg, ICC1, ICC2 before multilevel analysis.
Remove outliers in each group before any anslysis.
Careful interpretation of results needs relevant experience.

51. Extra readings
Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp (EI)
Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp (SSCI; FT50 Journal List)
Copyright (c) E.Y.Li
2019/11/14

52. Thank You for Listening
Q & A
Copyright (c) E.Y.Li
2019/11/14