Cultural Nuances, Assumptions and the Butterfly Effect: A Prelude A Brief Introduction to Generalizability Theory for the Uninitiated

What is Generalizability Theory? Generalizability Theory—the theory that deals with the degree to which one can say that conclusions drawn on the basis of data collected from sampling different domains can be applied with confidence to those domains.

Still Lost? What we’re talking about you best know as instrument reliability. Commonly known examples: inter-rater reliability, test-retest reliability, split-halves reliability, Kuder-Richardson reliabilities, Hoyt reliability, and Cronbach’s alpha

Test (True) Score Theory Observed scores are composed of true scores and error. The variance of the observed scores is partitioned. Then estimates are combined to produce a coefficient. X o = X t + X e X=score, t=true, o=observed, e=error S o 2 = S t 2 + S e 2 S 2 = variance r 11 = S t 2 /S o 2

True Score Theory and Generalizability Generalizabilty is an extension of true- score theory. Sources of Variation are viewed from different perspectives. S 2 O = S 2 LG + S 2 TG + S 2 LS + S 2 TS L=long term, T=temporary G=general, S=specific

So what does this have to do with Multivariate Analysis? All techniques we have studied deals with the degree of overlap of variance (information) for sets of variables (Tabachnick & Fidell, 2001). In the present case the overlap of interest is between observed variance for evaluators’ opinions about the spheres involved and the actual underlying dimensions influencing interpersonal interactions perceived by the evaluators--“true-score” variance. S 2 TOT = S 2 E + S 2 V + S 2 Sc + S 2 e E=evaluator, V=vignette,Sc=scale, e=error

Relationship of Gerneralizability to Multivariance To produce the variance estimates needed ANOVA is used. In this situation a 6 x 7 x 46 Evaluator (E) x Sphere + Primary Category (S) x Vignette (V) Mixed Effects Repeated Measures ANOVA is used. Evaluator and Vignette are considered random, Sphere fixed.

ANOVA DESIGN Doubly Repeated Measures Scale and Vignette Within Evaluator Between

Table 1 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Overall Evaluations Source of Variation SS df MS Vignette (V) Sphere (S) Evaluator (E) V x S V x E S x E V x S x E Error (e) Residual

Table 2 Variance Components and Expected Mean Squares for Overall Evaluations Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance Vignette (V).649  2 e + +  2 VE +  2 V.220 Sphere (S)  2 e + +  2 VS +  2 SE +  2 S ≥ Evaluator (E)  2 e + +  2 VE +  2 E V x S.687  2 e + +  2 VS ≤.207 V x E.429  2 e + +  2 VE ≤-.059 S x E  2 e + +  2 SE ≤ V x S x E  2 e +  2 VSE ≥.480 Error (e)  2 e ≥.480 Residual Total (TOT)  2 e +  2 S +  2 V +  2 E ≤ ≥  = S 2 S / S 2 TOT ≥ / = ≤  ≤ ≤ / = 0.886

Table 3 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Spheres Source of Variation SS df MS Vignette (V) Sphere (S) Evaluator (E) V x S V x E S x E V x S x E Error (e) Residual

Table 4 Variance Components and Expected Mean Squares for Spheres Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance Vignette (V).276  2 e + +  2 VE +  2 V.153 Sphere (S)  2 e + +  2 VS +  2 SE +  2 S ≥3.345 Evaluator (E)  2 e + +  2 VE +  2 E V x S.328  2 e + +  2 VS ≤.179 V x E.123  2 e + +  2 VE ≤-.026 S x E.991  2 e + +  2 SE ≤.842 V x S x E  2 e +  2 VSE ≥.149 Error (e)  2 e ≥.149 Residual Total (TOT)  2 e +  2 S +  2 V +  2 E ≤6.923 ≥7.072  = S 2 S / S 2 TOT ≥ / = ≤  ≤ ≤ / = 0.505

A Sample Calculation As an interesting example of how the  ’s are calculated here is one (see Table 2 for the data) Sphere (MS) =  2 e + +  2 VS +  2 SE +  2 S -V x S (MS) =.687  2 e + +  2 VS -S x E (MS) =  2 e + +  2 SE +Error (MS) ≤.480  2 e Sphere (Variance) ≤  2 S -Error (MS) ≤.480  2 e Sphere (Variance) ≥ Total (TOT) ≤  2 e +  2 S +  2 V +  2 E ≥  = S 2 S / S 2 TOT ≥ / = ≤ / = ≤  ≤ 0.882

The Conclusion What we get is three little numbers that are not as simple as they look: Primary Influence:  =.966 Spheres:  =.473 Overall:  =.879