1 Multilevel Models in Survey Error Estimation Joop Hox Utrecht University mlsurvey

2 Multilevel Modeling; some terminology/distinctions Two broad classes of multilevel models Multilevel regression analysis (HLM, MLwiN, SAS Proc Mixed, SPSS Mixed) Multilevel structural equation analysis (Lisrel 8.5, EQS 6, Mplus) Which are merging (Mplus, Glamm)

3 Multilevel Modeling; some terminology/distinctions Multilevel Modeling = A statistical model that allows specifying and estimating relationships between variables… … that have been observed at different levels of a hierarchical data structure Here mostly examples from multilevel regression modeling

4 Multilevel Regression Model Lowest (individual) level: Y ij =  0j +  1j X ij + e ij and at the Second (group) level:  0j =  00 +  01 Z j + u 0j  1j =  10 +  11 Z j + u 1j Combining: Y ij =  00 +  10 X ij +  01 Z j +  11 Z j X ij + u 1j X ij + u 0j + e ij

5 The Intercept-Only Model Intercept only model (null model, baseline model) Contains only intercept and corresponding error terms Y ij =  00 + u 0j + e ij Gives the intraclass correlation  (rho)  2 u  / (  e ² +  2 u0 )

6 The Fixed Model Only fixed effects for explanatory variables Slopes do not vary across groups Y ij =  00 +  10 X 1ij …  p0 X pij + u 0j + e ij Intercept variance U 0j across groups Variance component model Maximum Likelihood estimation, correct standard errors for clustered data

7 Using the Fixed Model in Survey Research? Multiple regression (including logistic) is a powerful analysis system (Jacob Cohen (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, ) Y ij =  00 +  10 X 1ij …  p0 X pij + u 0j + e ij Multiple regression model but correct standard errors for clustered data But…, most multilevel software does not correctly handle weights, stratification

8 Using the Fixed Model in Survey Research? Multilevel regression in survey data analysis: a niche product Individuals within groups Interviewer & Survey Organization effects Groups consisting of individuals Ratings & Measures of Contexts Occasions within individuals Longitudinal & Panel data

9 Individuals within groups Interviewer & Organization effects Potentially a three-level structure Respondents within Interviewers within Organizations Y ijk =   001 X ijk +  010 Z jk +  100 W k + u 0k + u 0jk + e ijk Variance components model

10 Interviewers in organizations “I am not selling anything” Split-run experiment on adding ‘not selling’ argument to standard telephone intro Multisite study: 10 market research organizations agreed to run experiment in their standard surveys Data from cases in 29 surveys within 10 organizations Predict cooperation rate Survey-level: experiment, saliency, special pop., nationwide, interview duration, length of intro before ‘not selling’ Organization level: no predictors, just variance component P ij =  00 +  01 Exp/Con ij +  02 X 1ij +…+  06 X 6ij + u 0j ( + e ij ) De Leeuw/Hox (2004). I am not selling anything: 29 experiments in telephone introductions. IJPOR, 16,

11 Interviewers in organizations across countries International cooperation on interviewer effects on nonresponse Data from 3064 interviewers, employed in 32 survey organizations, in nine countries Interviewer response rate, cooperation rate Standardized interviewer questionnaire (translated by organizations) Standardizing interviewer questionnaire across countries Not multilevel but multigroup SEM Confirmatory Factor Analysis shows comparable factors in (translated) questionnaires) Hox/de Leeuw (2002). The influence of interviewers' attitude and behavior on household survey nonresponse: an international comparison. In Groves, Dillman, Eltinge & Little (Eds.) Survey Nonresponse. New York: Wiley.

12 Predicting response rate Final multilevel model for interviewer response rates Predictor / ModelNull ModelFinal Model constant1.25 (.30).80 (.40) age.01 (.001) sex.05 (.02) experience.01 (.001 soc.val.-.02 (.01) foot in door.01 (.01)ns persuasion.10 (.01) voluntariness-.02 (.01) send other-.01 (.005)  ²country.59 (.37).58 (.36)  ²survey.41 (.13).39 (.12)

13 Multilevel analysis of Interviewer & Organization Effects Useful for methodological research Standard multilevel regression Response rates: logistic regression Estimation issues Discussed in Goldstein (2003), Raudenbush & Bryk (2004), Hox (2002) Currently best method Hox, de Leeuw & Kreft 1991; Hox & de Leeuw 2002; Pickery & Loosveldt 1998, 1999; Campanelli & O’Muircheartaigh 1999, 2002; Schräpler 2004;

14 Groups consisting of individuals Measuring contextual characteristics Aggregation: characterizing groups by summarizing the scores of individuals in these groups Contextual measurement: let individuals within groups rate group or environment characteristics What are the qualities of such ratings?

15 Measuring contextual characteristics Example: use pupils in schools to rate characteristics of the school manager 854 pupils from 96 schools rate 48 male + 48 female managers Variables: six seven-point items on leadership style Two levels: pupils within schools Pupils are informants on school manager Pupil level exists, but is not important

16 Measuring contextual characteristics Pupils in schools rate school managers Two levels: pupils within schools Analysis options Treat as two-level multivariate problem Multilevel SEM (Mplus, Lisrel, Eqs) Treat as three-level problem with levels variables, pupils, schools Multilevel regression (HLM, MLwiN)

17 Measuring the context with multilevel regression Three levels: variables, pupils, schools Intercept only model: Estimates: Intercept 2.57  2 school = 0.179,  2 pupil = 0.341,  2 item = 0.845

18 Measuring the context: Interpretation of estimates Intercept 2.57 Item Mean across items, pupils, schools  2 school = Variation of item means across schools  2 pupil = Variation of item means across pupils  2 item = Item variation (inconsistency)

19 Measuring the context: Reliability of measurement Decomposition of total variance over item, pupil & school level Pupil level reliability Consistency of pupils across items Idiosyncratic responses, unique experience  pupil =  2 pupil /(  2 pupil +  2 item /k)  pupil = 0.71

20 Measuring the context: Reliability of measurement Decomposition of total variance over item, pupil & school level School level reliability Consistency of pupils about manager  school = 0.77

21 Measuring the Context: Increasing reliability School level reliability depends on Mean correlation between items Intraclass correlation for school Number of items k Number of pupils n j  goes up fastest with increasing n j

22 Measuring the context: Combining information Assume school managers are rated on these 7 items by pupils and themselves Three levels: items, pupils, schools Two dummy variables that indicate pupil & self ratings Variances item (1), pupil (1), school (2 + cov) Item variance (error) Pupil variance (bias) Manager variance (systematic) Rating covariance (validity)

23 Example: Measuring neighborhood characteristics Neighborhoods & Violent Crime Assessment of neighborhoods 343 neighborhoods ± 25 respondents per neighborhood interviewed & rated own neighborhood (respondent level) Ratings aggregated to neighborhood level Census information on neighborhood added Sampson/Raudenbush/Earls (1997). Neighborhoods and violent crime: A multilevel study of collective efficacy. Science, 277,

24 Example: Measuring neighborhood characteristics Ratings aggregated to neighborhood level At lowest level demographic variables of respondents added to control for rating bias due to different subsamples Neighborhood ratings aggregated conditional on respondent characteristics Y ijk =   001 X ijk + u 0k + u 0jk + e ijk Intercept-only + individual covariates

25 Occasions within individuals Six persons on up to four occasions Lowest level: occasion; Second: person Mix time variant (occasion level) and time invariant (person level) predictors Time: trend covariate (1, 2, 3…) or occasion dummies (0/1) Missing occasions are no problem

26 Longitudinal data: Occasion level Occasion level, time indicator T Y ti =  0j +  1j T ti + e tj Intercept and slope coefficients vary across the persons They are the starting points and rates of change for the different persons Use  for occasion level coefficient, and t for the occasion subscript On person level we have again  and i

27 Longitudinal data: Multilevel model Occasion level:Time varying covariates Y ti =  0i +  1i T ti +  2j X ti + e tj Person level: time invariant covariates  0j =  00 +  01 Z i + u 0i  1j =  10 +  11 Z i + u 1i  2j =  20 +  21 Z i + u 2i T time-points, at most T-1 time varying predictors Or T time varying predictors and no intercept

28 Longitudinal data: NLSY Example Subset of National Longitudinal Survey of Youth (NLSY) 405 children within 2 years of entering elementary school 4 repeated measurement occasions Child’s antisocial behavior and reading recognition skills 1 single measure at 1 st occasion Mother’s emotional support and cognitive stimulation

29 NLSY Example: Linear Trend Multilevel regression model for longitudinal GPA data No ‘intercept-only’ model, start with a model that includes time Occasion fixed Antisoc tj =  00 +  10 Occ ti + u 0i + e ti Occasion random Antisoc tj =  00 +  10 Occ ti + u 1i Occ ti + u 0i + e ti Different individual trends over time

30 NLSY Example: Results linear trend Linear, FixedLinear, Random Intercept1.58 (.11)1.56 (.10) Occasion0.14 (.03)0.15 (.04)   intercept 1.84 (.17)0.96 (.31)   occasion (.04)  intercept,occasion -.09 (.10) ee 1.91 (.09)1.74 (.10) Deviance

31 Complex Covariance Structures Standard model for longitudinal data Occasion random: Antisoc tj =  00 +  10 Occ ti + u 1i Occ ti + u 0i + e ti Variance components:  e 2 and  00 2 Assumes a very simple error structure Variance at any occasion equal to  e 2 +  00 2 Covariance between any two occasions equal to  00 2 Thus, matrix of covariances between occasions is

32 Complex Covariance Structures Multivariate multilevel model No intercept, include 6 dummies for 6 occasions No variance component at occasion level All dummies random at individual level Equivalent to Manova approach to repeated measures Covariance matrix: Add occasion, fixed

33 Complex Covariance Structures Restricted model for longitudinal data Specific constraints on covariance matrix between occasions Example: assume that autocorrelations between adjacent time points are higher than between other time points (simplex model) Example: assume that autocorrelations follow the model e t =  e t-1 +  Add occasion, fixed or random

34 NLSY Example: Linear trend, Complex covariance structure 1. Occasion fixed, unrestricted covariance matrix across occasions 2. Occasion fixed, covariance matrix autocorrelation structure 3. Occasion random, covariance matrix autocorrelation structure

35 NLSY Example: Results linear trend, fixed part Fixed, Un- constrained Fixed, Auto- correlation Random, Autocorrelation Intercept 1.55 (.10)1.54 (.13) Occasion 0.14 (.04)0.15 (.05).15 (.05) Deviance Linear trend + random slope model deviance with 8 less parameters  2 =14.2, df=8, p=0.08 Far worse than unconstrained model  2 =97.7, df=8, p<0.0001

36 NLSY Example: Results linear trend, random part Fixed, Un- constrained Fixed, Auto- correlation Random, Autocorrelation Occasion linear --Aliased out (redundant) Occasion dummies Full covariance matrix, all elements significant Diagonal variance, autocorr. rho both significant

37 Advantages of Multilevel Modeling Longitudinal Data Missing occasion data are no problem Manova = listwise deletion, which wastes data Manova = Missing Completely At Random (MCAR) Multilevel model = Missing At Random (MAR) Can be used for panel & growth models Rate of change may differ across persons, and predicted by person characteristics Easy to extend to more levels (groups)

38 References for Multilevel Analysis J.J. Hox, Applied Multilevel Analysis. ( (introductory) J.J. Hox, Multilevel Analysis. Techniques and Applications. Hillsdale, NJ: Erlbaum. (intermediate) T.A.B. Snijders & R.J. Bosker (1999). Multilevel Analysis. Thousand Oaks, CA: Sage. (more technical) H. Goldstein (2003). Multilevel Statistical Models. London: Arnold Publishers. (very technical)

39