The Structural Equation Modeling approach

The Structural Equation Modeling approach
The effect of survey characteristics on random and systematic errors in surveys by Willem Saris

college titel en nummer
Designing questions Questionnaire design consists of a series of decisions: topic, concept, form of the question, response categories etc. (Andrews 1984). If we know the consequences of our decisions questionnaire design can be done in a scientific way This idea is the basic motivation of our long term dedication to our research on the effects of survey characteristics on data quality 21/11/18 college titel en nummer

An Illustration Three ESS questions about satisfaction: On the whole how satisfied are you with the present state of the economy in Britain ? Now think about the national government. How satisfied are you with the way it is doing its job ? And on the whole, how satisfied are you with the way democracy works in Britain ? 21/11/18 college titel en nummer

Three alternative response scales
A first method 1 Very satisfied , 2 fairly satisfied, 3 fairly dissatisfied or 4 very dissatisfied A second method very very dissatisfied Satisfied A third method: 1 not at all satisfied 2 satisfied 3 rather satisfied 4 very satisfied 21/11/18 college titel en nummer

Does it matter what we choose ?
The combination of traits and scales gives 3x3 different questions. All 9 questions have been presented to a British sample of 485 people That means that differences in responses can not be due to sampling fluctuation but only question format 21/11/18 college titel en nummer

It matters for the distributions
21/11/18 college titel en nummer

It matters for the correlations

Research questions Using the MTMM design one can see that there are random and systematic errors We have many examples of the same kind How large are the different errors ? Can we derive general rules with respect to the effects of question characteristics ? 21/11/18 college titel en nummer

Content of this presentation
1. Definition of reliability and validity in terms of random and systematic error 2. SEM approaches to estimation of reliability and validity 3 The effects of survey characteristics on reliability and validity 4 Prediction of data quality by SQP 5 Conclusions 21/11/18 college titel en nummer

Definition of reliability and validity
Definition of reliability and validity in terms of random and systematic error

The Classical test model
y is the observed variable e is random measurement error across persons and occasions t is called the true score: t = y - e The strength of the relationship between t and y is called the reliability The reliability = 1 - var(e) 21/11/18 college titel en nummer

Difference due to the method used
The strength of the relationship between t and f is called the validity: The Validity = 1 - var (m) 21/11/18 college titel en nummer

Measurement model for two traits , same method

SEM Approach to estimation of survey errors
Different Designs and Models

Different designs to estimate reliability and validity
Different designs to estimate the reliability and validity: test retest design quasi simplex design Split ballot design Evaluation of validity by nomologic network Classic MTMM design SB MTMM design RSB MTMM design 21/11/18 college titel en nummer

Test retest design and model
The same observation is done twice with the same method Assuming that there is no difference between f and t the relationship between f and y is equal to the square root of the reliability if ri1 = ri2 then r(yi1yi2) = ri12 so the reliability = r(yi1yi2) but 21/11/18 college titel en nummer

Criticism on the Test retest model
This model reduces only to the test retest model if 1.no change in opinion between the first and the second measurement 2.no memory effects 3.no method effects 4.equal reliability for the different measures of the same trait. How to combine 1 & 2 ? 21/11/18 college titel en nummer

The quasi simplex design and model
In this design the same observation is repeated 3 times with the same method Now the variable on interest can change and the reliability coefficient can still be estimated but 21/11/18 college titel en nummer

Criticism on the Quasi simplex model
It is a lag one model. An effect of f1 on f3 is impossible and will lead to completely different estimates of the reliabilities (Coenders) All temporary effects, not present on the next moment are included in the measurement error term which will be overestimated (van der Veld) 21/11/18 college titel en nummer

Split Ballot Design In many studies (Schuman and Presser 1981) the sample is randomly split up in two or more groups Each group is confronted with a different form of the question If the score on f is known from other sources the error = yi - fi and the estimate of bias = mean(y) - mean(f) If the score on f is unknown one can obtain an estimate of the relative bias: rel. bias = mean(ym1) - mean (ym2) 21/11/18 college titel en nummer

Criticism on the Split ballot design
If the relative bias is significantly different from zero then we know that at least one of the methods in not valid but we don’t know which one. In order to see which method is better, the relationships with variables, that should correlate highly with the variable of interest, are observed The method that generates the highest correlation is the most valid measure but 21/11/18 college titel en nummer

Criticism of the construct validity approach
Construct validity is normally evaluated by estimating the correlation with other correlated variables ry1j,xi = r1j v1j r and ry2j,xi = r2j v2jr The difference in correlation can come from the reliability or the validity So v1j and v2j should be compared not the correlations 21/11/18 college titel en nummer

Classical MTMM design and model
Andrews (1984) suggested the Classical MTMM model for the design with 3 traits and 3 methods of Campbell and Fiske (1956) In that case the reliability and validity coefficients for all 9 questions and the correlations between the traits can be estimated using a factor model Saris and Andrews (1991) suggested the True Score MTMM model 21/11/18 college titel en nummer

Factor model representation

Criticism on the MTMM design
Respondents have to answer three times a question about the same topic After 20 minutes most people have forgotten what they answered before (Van Meurs and Saris) But this requires long questionnaires The response burden is high: it may be that people become less concentrated It is also possible that they learn how to answer these questions 21/11/18 college titel en nummer

The Split Ballot MTMM design
The Split Ballot MTMM design has been developed by Saris, The estimation and testing are described in Saris, Satorra and Coenders (2004) The idea is that the sample is split up randomly in two or more groups Each group gets only 2 forms of the questions Using different groups all methods will be used at least once 21/11/18 college titel en nummer

Missing data and Identification in the two groups design
Samples 1 + 2 Sample 2 Sample 1 Method 3 none Method 2 Method 1 2group design Matrix for a Covariance Although one block of variables is completely missing by design all the parameters of the model can still be estimated unless: The correlations between the traits are zero or The correlations are exactly equal to each other 21/11/18 college titel en nummer

Criticism of the SB-MTMM design
Large samples are needed to get estimates with the same precision as the classical MTMM design A more serious problem of all MTMM designs is that the method and occasion effects are confounded. We can not separate these effects without an exact repetition of the same method at two occasions. 21/11/18 college titel en nummer

The Repeated SB-MTMM design
From this model follows: (Y1jk, Y2jk) = r1jkv1jk (F1,F2)v2jkr2jk r1jkm1jkm2jkr2jk r1jko1jko2jkr2jk To make a distinction between the effect of the methods and the occasions exact repetition is required 21/11/18 college titel en nummer

Results of the MTMM experiment on satisfaction

The explanation of the different correlations for satisfaction
For method 1(a 4 point scale) this correlation was .481 For method 2 (an 11 point scale) this correlation was .626 The correlation between the observed variables can be predicted with the MTMM models as follows: r(Y1j,Y2j) = r1jv1j r(f1,f2)v2jr2j + r1jm1jm2jr2j 21/11/18 college titel en nummer

Alternative models Alternative models for correlated errors in stead of method effects: Multiplicative models Memory effect models Acquiescence model Variation in response functions model Corten et al (2002) and Saris and Albers (2003) have shown that the MTMM model is better than the other models. 21/11/18 college titel en nummer

Conclusions The best design and model to estimate reliability and validity is the MTMM, preferably the RSB MTMM design and model The results cannot be generalized to other measures This approach is rather expensive if all researchers should use this model to correct for measurement error Therefore an alternative has been developed. 21/11/18 college titel en nummer

Prediction of Survey quality
Meta analysis of MTMM experiments

An alternative: Meta analysis of MTMM experiments
Andrews (1984) performed a meta analysis of MTMM experiments The MTMM studies provide estimates of the reliability and validity of questions The questions can be coded on design decisions Regressing the quality criteria on the design decisions based on a large number of questions provides estimates of the effects of the design decisions on the question quality 21/11/18 college titel en nummer

Meta analysis C represents the score on a quality criterion Dij represents the dummy variables for the jth nominal variable. All dummy variables have the value zero unless the specific characteristic applies for a question. Continuous variables, like Ncat, have not been categorized The intercept (a) is the reliability or validity of the instruments if all variables have a score of zero. 21/11/18 college titel en nummer

The most recent cross national meta analysis
In total, 87 MTMM studies have been used containing 1067 survey items. They come from : Andrews (1984) and Rogers, Andrews and Herzog (1989) in the US. Költringer (1995) in Austria Scherpenzeel and Saris (1997) in the Netherlands Billiet and Waege in Belgium 21/11/18 college titel en nummer

The most recent cross national meta analysis
The results have been presented in full in Saris, Van der Veld and Gallhofer (2004) The explained variance for reliability = .47 The explained variance for validity = .61 Some results of the analysis are presented 21/11/18 college titel en nummer

Meta analysis 21/11/18 college titel en nummer

Conclusion The meta analysis of the MTMM experiments gives an explanation of the variation in reliability and validity on the basis of the choices made in designing a survey (question) This result contains all the present information that is available from MTMM experiments This is a temporary result because new data are collected. 21/11/18 college titel en nummer

Survey Quality Predictions
the program SQP

Survey Quality Predictions
When a question is coded on all the characteristics the regression equation from the meta analysis can be used for the prediction of the reliability and validity. Reliability = N I H B L OS W EB F IM FE FB CC + etc. 21/11/18 college titel en nummer

Survey Quality Prediction
It will be clear that the coding and the computation is a lot of work. We have made a semi automatic program for the prediction of the quality of questions SQP 21/11/18 college titel en nummer

Survey Quality Prediction
These predictions are based on 1067 questions from 87 experiments Predictions are made for questions in three languages: English, German and Dutch The ESS contains in each wave 6 MTMM experiments in 22 countries: more than 1000 questions A new version of SQP will be based on more than 5000 questions and predict data quality for questions in more than 20 languages. 21/11/18 college titel en nummer

General Conclusions The best way to estimate reliability and validity is the MTMM design and model Use of this design in all survey research would be rather expensive and too difficult The meta analysis of many of these MTMM experiments provides estimates of the effects of the design choices on the data quality These results can be used to predict the quality of questions while researchers do not have to collect new or extra data 21/11/18 college titel en nummer

General Conclusions These predictions can be used for: improvement of questions before they are used in practice estimation of the quality of composite scores after data have been collected correction for measurement error after data have been collected 21/11/18 college titel en nummer

The Structural Equation Modeling approach

Similar presentations

Presentation on theme: "The Structural Equation Modeling approach"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

The Structural Equation Modeling approach

Similar presentations

Presentation on theme: "The Structural Equation Modeling approach"— Presentation transcript:

Similar presentations

About project

Feedback