Multilevel Model Analysis: Method and Applications Eldon Y. Li University Chair Professor Department of MIS National Chengchi University, Taiwan http://www.calpoly.edu/~eli *** All right reserved. Reference to this document should be made as follows: Li, E.Y. “Multilevel Model Analysis: Method and Applications”, unpublished lecture, National Chengchi University, 2018 *** Copyright © E.Y.Li 2019/11/14

Agenda About me Abstract Introduction Why multilevel model? When multilevel model? How multilevel model? Application example 1 Application example 2 Copyright (c) E.Y.Li 2019/11/14

About Me Eldon Y. Li is a university chair professor and former department chair of MIS at National Chengchi University, an adjunct chair professor of Asia University in Taiwan, and a former professor and coordinator of the MIS program at College of Business, California Polytechnic State University, San Luis Obispo, California, USA. He was the dean of College of Informatics and the director of Graduate Institute of Social Informatics at Yuan Ze University in Taiwan, as well as professor and founding director at the Graduate Institute of Information Management at the National Chung Cheng University in Chia-Yi, Taiwan. He received his PhD from Texas Tech University in 1982. He is the editor-in-chief of several international journals. He has published more than 250 papers in various topics related to innovation and technology management, human factors in information technology (IT), strategic IT planning, software quality management, and information systems management. His papers appear in Journal of Management Information Systems, Research Policy, Communications of the ACM, Internet Research, Expert Systems with Applications, Computers & Education, Decision Support Systems, Information & Management, International Journal of Medical Informatics, Organization, among others. 2019/11/14

Multilevel Model Analysis: Method and Applications Abstract Conventional survey studies usually collect individual data from different groups and analyze them independently in each group, unless there is no significant group difference. In contrast, multilevel model analysis allows individual data with organizational differences to be included in one regression model by treating these differences as higher-level independent variables.  Such kind of model is known as hierarchical linear model (HLM). This lecture introduces various research models and discusses why, when, and how to perform multilevel model analysis. The applications of multilevel model analysis are elucidated using two published studies in information systems field. Copyright (c) E.Y.Li 2019/11/14

Introduction – Model Analysis Relational model Causal model Behavioural model Process model Mediation model Moderation model Moderated mediation model Multilevel model Mixed model

Relational model (Prediction model) Source: Li, E.Y.* (1994) "Artificial Neural Networks and Their Business Applications," Information & Management (Elsevier), Vol. 27, No. 5, October, pp. 303-313.

Relational model (Prediction model) Yen/Dollar Exchange (-1) Nikkei 225 Yen/Dollar Exchange (-1.5) Topix Composite Yen/Dollar Exchange (-2) Topix Large Yen/Dollar Exchange (-3) Topix Small Source: Li, E.Y.* and Soenen, L. (1994) "Dollar Value of the Yen and Stock Price Reactions in Japan," Journal of Global Business (U.S.A.), Vol. 5, No. 1, Spring, pp. 5-12.

Causal model Source: Wu, Y.L., Li, E.Y.*, and Chang, W.L. (2016) "Nurturing user creative performance in social media networks: an integration of habit of use with social capital and information exchange theories," Internet Research (Emerald), Vol. 26, No. 4, pp.869-900. (SSCI)

Behavioral model Theory of Reasoned Action Source: Fishbein, M., & Ajzen, I. (1975). Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley. 

Process model Source: Bhattacherjee, A. and Premkumar, G. (2004). Understanding changes in belief and attitude toward information technology usage: a theoretical model and longitudinal test. MIS Quarterly, 28 (2), 229-254.

Partial mediation model Source: Huang, Y.H.*, Li, E.Y., and Chen, J.S. (2009.3) "Information Synergy As the Catalyst Between IT Capability and Innovativeness: Empirical Evidence from Financial Service Sector," Information Research: An International Electronic Journal, Vol. 14, No. 1, pp. 1-11. Figure 1: Research Model

Full mediation model Source: Yen, H.J.R., Li, E.Y. and Niehoff, B. (2008.9). Do organizational citizenship behaviors lead to information system success? testing the mediation effects of integration climate and project management. Information & Management, 45 (6), 394-402.

Moderation model Source: Venkatesh, V., Morris, M.G., Davis, G.B., and Davis, F.D. (2003). User Acceptance of information technology: Toward a unified view. MIS Quarterly, (27:3), 425-478.

Moderated mediation model Source: Ajzen, I. “TPB Diagram”, available at http://people.umass.edu/aizen/tpb.diag.html

Multilevel model Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307.

Multilevel model Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp. 144-178.

Mixed model Source: DeLone, W. and McLean, E. (2003). The DeLone and McLean Model of information systems success: a ten-year update. Journal of Management Information Systems, 19 (4), 9-30.

Mixed model Source: Wixom, B.H. and Todd, P.A. (2005). A theoretical integration of user satisfaction and technology acceptance. Information Systems Research, 16 (1), 85-102.

Why multilevel model? Company 1 Company 2 Company 3 Company 4

When multilevel model? Condition for applying multilevel analysis Variance between companies  relatively large Variance within a company (between employees)  relatively small. If variance between companies is small, use single level analysis. If variance within a company (between employees) is large, remove this company from the data.

When? (Cont.) Intraclass agreement index (rwg) Also called Within-organization agreement index rwg (J) value indicates the degree to which the responses to a measurement scale by members of the same organization converge. rwg (J) value >0.70 (James et al., 1984) James LR, Demaree RG, Wolf G (1984) Estimating within-group interrater reliability with and without bias. J Appl Psychol 69:85response

When? (Cont.) Intraclass correlation coefficients ICC1 compares the between-organizations variance with the within-organization variance to indicate the portion of variance in individual responses (MSW) accounted for by the between-organizations difference (MSB). > 0.12 (Bliese 2000) K = the average sample size from a company Bliese, P.D. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K.J. Klein and S.W.J. Kozlowski (eds.), Multilevel Theory, Research, and Methods in Organizations. San Francisco: Jossey-Bass, 2000, pp. 349–381.

When? (Cont.) Intraclass correlation coefficients (cont.) ICC2 reveals the reliability of the mean of an organization-level variable. If low reliability, multilevel analysis is needed. That is, MSB should be large, MSW should be smaller and no more than than 40% of MSB. > 0.60 (Bliese 2000)

How multilevel model? Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307.

How? (Cont.)

Application example 1 Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307. Copyright © E.Y.Li 2019/11/14

Application example 1 - Originality The IS Success (ISS) model of DeLone and McLean (1992, 2003) IS’s Qualities User’s Satisfaction Organization’s Net Benefits IS Developers Users ??? User Managers Copyright © E.Y.Li 2019/11/14

Originality (Cont.) The ISS model of DeLone and McLean (1992, 2003) prescribes IS’s quality (including information, system, and service qualities), user’s satisfaction, and organization’s net benefits as the three integrated components. While IS developer is responsible for IS quality, users are concerned with satisfaction, leaving net benefits unattended.

Research questions What are the factors influencing net benfits of user department’s IS appcliation (as user manager’s job performance)? What are the interaction effects of these factors on net benefits?

Underlying theories Copyright © E.Y.Li 2019/11/14

Research model Source: Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307. Copyright © E.Y.Li 2019/11/14

Solution model

Solution model UDISPij represents the ith individual score of UDISP in jth organization. TMSj represents the aggregate score of TMS in jth organization. ISKij represents the ith individual score of ISK in jth organization. ISAij represents the ith individual score of ISA in jth organization. γkl represents the slope of the kth level-1 predictor interacting with the lth level-2 predictor. Ukj is a normal distribution and represents the residual of slope of kth level-1 predictor in the jth organization. rij is a normal distribution and represents the residual of regression model in individual level.

HLM software for Windows

Model specification in HLM Level-1 Level-2 Coefficients Predictors ---------------------- --------------- INTRCPT1, B0 INTRCPT2, G00 TMS, G01 ISK slope, B1 INTRCPT2, G10 TMS, G11 ISA slope, B2 INTRCPT2, G20 TMS, G21 Summary of the model specified (in equation format) --------------------------------------------------- Level-1 Model Y = B0 + B1*(ISK) + B2*(ISA) + R Level-2 Model B0 = G00 + G01*(TMS) + U0 B1 = G10 + G11*(TMS) + U1 B2 = G20 + G21*(TMS) + U2

Method - measures Job performance  User department’s IS performance. Opportunity  Top management support. Capability  User manager’s knowledge about IS applications. Willingness  User manager’s attitude toward IS applications. Legend:  =measured by.

Method - subjects Data requirements: Using similar information systems: ERP In similar industry: Manufacturing System experience: At least 1 year At least 10 companies in the industry: 42 At least 5 data ponits (departments) in each company: Average 6-7 deparments, Total 283

Analysis Focus group was used to ensure face validity of the survey questionnaire. Valid survey data were collected from 283 user managers and 42 top managers of 42 different Chinese manufacturing companies in which ERP systems were being utilized. Each company sample=6 to 7  K=6; ICC1=0.253 >0.12; ICC2=0.670 >0.60. The model can be validated by using Hierarchical Linear Modelling (HLM) software.

Results

Example 1 Conclusions Top management support, user-manager knowledge, and user-manager attitude all affect the level of UDISP significantly. (H1, H2, H3 supported) Top management support significantly moderates the relationship between user-manager attitude and UDISP. (H5 supported) The interaction effect of top management support and user-manager knowledge on UDISP is not significant. (H4 not supported)

Application example 2 Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp. 144-178.

Solution model Department-level Model: Employee’s Loyal Use: ELUij = β0j + β1j (EPBij ) +β2j (EPWij) +rij, Company-level Model: β0j = γ00 + γ01 (OLSQj) + γ02 (OLIQj) + γ03 (OLSOCBj) +U0j β1j = γ10 + γ11 (OLSQj) + γ12 (OLIQj) + γ13 (OLSOCBj) +U1j β2j = γ20 + γ21 (OLSQj) + γ22 (OLIQj) + γ23 (OLSOCBj) + U2j The final formula is as follows: ELUij = γ00 + γ01 (OLSQj) + γ02 (OLIQj) + γ03 (OLSOCBj) +U0j +(γ10 + γ11 (OLSQj) + γ12 (OLIQj) + γ13 (OLSOCBj) +U1j) EPBij +(γ20 + γ21 (OLSQj) + γ22 (OLIQj) + γ23 (OLSOCBj) + U2j)EPWij +rij,

Solution model ELUij represents the ith individual score of ELU in jth organization. OLSQj represents the aggregate score of OLSQ in jth organization. OLIQj represents the aggregate score of OLIQ in jth organization. OLSOCBj represents the aggregate score of OLSOCB in jth organization. EPBij represents the ith individual score of EPB in jth organization. EPWij represents the ith individual score of EPW in jth organization. γkl represents the slope of the kth level-1 predictor interacting with the lth level-2 predictor. Ukj is a normal distribution and represents the residual of slope of kth level-1 predictor in the jth organization. rij is a normal distribution and represents the residual of regression model in individual level.

Underlying theories Individual Level Organization Level Interactionism Paradigm Situational Strength Theory Employee’s Perceived Workload Social Information Processing Theory Social Learning Theory Rational Choice Theory Cost–Benefit Analysis Loyal Use Individual Perceptions Causal relation Construct Theoretical foundation Bottom-up/Top-down process Copyright © E.Y.Li 2019/11/14

IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02 Results Individual Level OL-IQ OL-SOCB Employee’s Perceived Workload Employee’s Loyal Use Organization Level OL-SQ H1 -0.11* ns -0.12* H2 0.32*** 0.47*** Perceived Benefits IL-IQ 0.11* -0.29*** IL-SQ IL-SOCB IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02 0.14*   -0.07*** 0.23***

Results

Results

Results

Example 2 Conclusions IL-SQ, IL-IQ, IL-SOCB all affects positively EPB. Only IL-SQ affects negatively EPW. IL-SQ and IL-IQ affects positively ELU. EPB affects positively ELU. EPW affects negatively ELU. OL-SQ and OL-SOCB moderates negatively EPBELU. OL-IQ moderates positively EPWELU.

Overall Conclusions Multilevel model analysis overcomes the group differences and analyzes samples from multiple groups in one regression model. When group differences are not significant, use single level analysis; when significant, use multilevel analysis. Test rwg, ICC1, ICC2 before multilevel analysis. Remove outliers in each group before any anslysis. Careful interpretation of results needs relevant experience.

Extra readings Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307. (EI) Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp. 144-178. (SSCI; FT50 Journal List) Copyright (c) E.Y.Li 2019/11/14

Thank You for Listening Q & A Copyright (c) E.Y.Li 2019/11/14