1. Funded through the ESRC’s Researcher Development Initiative Department of Education, University of Oxford Session 3.3 & 3.4: Teacher Expectancy Example

3. (Meta-analysis data from Raudenbush & Bryk, 2002)

4.  Do teacher expectations influence student IQs?  Teachers led to have high expectations of experimental (through bogus feedback) but not control students.  The focus is on the effect of how long teachers knew students prior to the experimental intervention.

5. Teacher Expectancy Effects on IQ (Meta- analysis data from Radudenbush & Bryk, 2002 Do teacher expectations influence student IQs? Teachers led to have high expectations of experimental (through bogus feedback) but not control students. Focus here is on the effect of how long teachers knew students prior to the experimental intervention.

6. 6 Teacher Expectancy Effects on IQ (Meta- analysis data from Radudenbush & Bryk, 2002

7. Median75 th %tile25 th %tile90 th %tile10 th %tile Potential Outliers

8. Mean effect size and homogeneity analyses

10. 2523 0 *-------------------------------------------------------------- 2524 0 *' Macro for SPSS/Win Version 6.1 or Higher 2525 0 *' Written by David B. Wilson (dwilson@crim.umd.edu) 2526 0 *' Meta-Analyzes Any Type of Effect Size 2527 0 *' To use, initialize macro with the include statement: 2529 0 *' INCLUDE "[drive][path]MeanES.SPS". 2530 0 *' Syntax for macro: 2532 0 *' MeanES ES=varname /W=varname /PRINT=option. 2534 0 *' E.g., MeanES ES = D /W = IVWEIGHT. 2535 0 *' In this example, D is the name of the effect size variable 2536 0 *' and IVWEIGHT is the name of the inverse variance weight 2537 0 *' variable. Replace D and INVWEIGHT with the appropriate 2539 0 *' variable names for your data set. 2540 0 *' /PRINT has the options "EXP" and "IVZR". The former 2541 0 *' prints the exponent of the results (odds-ratios) and 2542 0 *' the latter prints the inverse Zr transform of the 2543 0 *' results. If the /PRINT statement is ommitted, the 2545 0 *' results are printed in their raw form.

12. Conclusions: Small (NS) effect size based on both Fixed & Random models. Significant unexplained variance suggesting non-generalisability of effects across studies and the need for a random effects model.

13. Analogue to the ANOVA analyses

14. Conclusions: Large effect of weeks (20.4/35.8= 57% var expl); Total Residual, variance component & residual by group all NS; ES significant for 1 st two groups, NS last two groups

15. Regression analyses

16. Conclusions: Large effect of weeks (54% var expl); Constant term highly significant (at intercept = 0); Residual variance NS (variance component = 0)

17. Conclusions: Large effect of weeks (21% var expl); Constant term highly significant (at intercept = 0); Residual var NS; Does not do as well as categorised weeks

18. Variations of the previous analyses

19. Conclusions: Small, NS effect; Resid var marginally significant for “Aware” not “blind”

20. Conclusions: Small, marginally significant effect (4.1/34.3=12% var expl); ES NS for “Group” but signi for “individual” Resid var for group signif but individual NS; All effects very small

21. Conclusions: Small, marginally significant effect (12% var expl); Constant term (Group) NS; Resid var signif (but variance component NS)

22. Conclusions: Effect of test type no longer signif when weeks included. Effect of weeks nearly unaffected. Note can only look at multiple variables with regression.

24. Website Address to get MLwiN Harvey Goldstein developed the MLwiN statistical package used here and has made many contributions to multilevel modeling, including meta-analysis.

25. Always a bit dangerous to say some one person invented a new approach. However, fair to say that Stephen Raudenbush at least popularised the multilevel approach to meta-analysis with the meta-analysis of the teacher- expectancy data considered here. Raudenbush, S.W. and Bryk, A.S. (2002).Hierarchical Linear Models (Second Edition).Thousand Oaks: Sage Publications, 482 pp. Raudenbush, S.W. (1984). Magnitude of teacher expectancy effects on Pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments.Journal of Educational Psychology, 76, 1, 85-97.

26. In an empty MLwiN file, puts the xx input variables into first xx columns. (can also add new data to existing files). Check to see that data is correct and click on “paste” button

27. Check the MLwiN “names” file to see that data looks ok (e.g., missing values; min & max values).

28. MLwiN will open an empty equation that you have to construct. Click on the “y” to bring up this screen. select “d” (the effect size) as the dependent variable Select “2” for “N of levels” select “ID” for Level 1 select “d” for Level 2

29. 1.Click “Add Term” Button (bottom equations window) 2.Select “cons” (variable = 1 for all cases) 3.Click the “done” button 1 2 3

30. 1.Click “Cons” in the equation 2.Tick “Fixed Parameter” & “j(id)” but not “i(d)” 3.Click the “done” button 1 2 3

31. 1.Now click “add term” button 2. This will bring up the “X-Variable” select SE (the standard error computed earlier) 3.Tick only the “i(d)” box 4. Click “done” 1 2 3 4

32. Now we want to constrain the variance at level 1 to be fixed at 1.0. Under “model” select “constrain parameters”; will bring up “parameter constraint” window 1 1

33. In the parameter constraint window: 1.Click the “random” button 2.Change “d: SE/SE” to 1 3.Change “to equal” to 1” 1 2&3

34. 1.“store” the constraints in the first empty column (“C19”) 2.Click the “attach random constraints” button. 3. Close the “Parameter Constraint” Window 1 2 3

35. After Closing the “parameter constraint” window (last slide) Click on “start” button in “equation” window (may have to click estimates button to get values). Compute chi-square value in command interface window Conclusion: The mean effect size (.078) is not significant. The chi-square is significant; there is study-to-study variation. Reasonable to explore moderator variables

36. Conclusion: The effect of weeks (-.013/.005) is significant The mean effect size (.162/.055) signif (when weeks = 0). chi-sq signif; some remaining study-to-study variation.

37. Conclusion: categorized weeks does best of of (chi-sq = 16.568)

38. Conclusion: Main Effect of Aware vs. Blind is NS For a categorical variable, you choose a reference (“left out” category. Default is the 1 st category >pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 35.608 ->cprob b1 17 = 0.0051714

39. Conclusion: Effect for Blind vs. Aware reduced by controlling for “wkcat” but was already nonsignificant For a categorical variable, you choose a reference (“left out” category. Default is the 1 st category >pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 16.445 ->cprob b1 17 = 0.49254

40. Conclusion: Effect seems larger for individually administered tests, but not after control for weeks (wkcat)

41. Order = 1 to specify a 2-way interaction term Variables in interaction Conclusion: No interaction effect (chi-sq little different than wkcat alone (15.114 vs. 16.568). Notice that the effect of test type (and its SE) are very large (.2901/.4859=.597)

42. Grand Mean Centered Conclusion: Same results but estimated & SE for individual term is smaller (reduced multicollinearity by grand mean centering the wkcat variable). Note that chi-sq is the same. Grand Mean Centered

44. Conclusion: The effect of weeks (-.013/.005) & chi-sq (28.937) same as with original weeks. The mean effect size (.136/.049) signif (when weeks = 2). Weeks is centered at 2. >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115

45. Conclusion: Constant term (intercept weeks = 6) is signif (.083/.042=1.98) Constant term (intercept weeks = 7) is NS (.070/.042=1.68) >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115 Weeks centered at 6 Weeks centered at 7

46. Conclusion: The linear term is significant but the quad term is not. >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 26.237; ->cprob b1 16 = 0.050779

47. Conclusion: All three polynomial terms are significant and the residual variance component is substantially reduced.

48. Conclusion: The linear term based on the log transform explains more variance than the original (untransformed) weeks (chi-sq = 24.636 vs. 28.937).

49. Conclusion: Intercept at wkcat = 2 is significant, but intercept at wkcat = 3 is not

50. 1 Caterpillar plot based on L1 residuals. Go to the “model” menu and select “residuals” option. This will bring up the “settings” window. Set “SD (comparative)” to 1.96; 3. Set “level” to “1d”; 4. click the “Calc” button; 5. click on the “plot” button to bring up the next window. In the “plot” window select “residual +/- 1.96SD x rank. This brings up the original graph. Clicking on the graph bring up a window to modify the graph (a bit)

52.  The mean effect size associated with the intervention was not significant. However, the results did not generalise across studies (there was study-to-study variation).  Consistent with a priori predictions, the effect size was significantly moderated by the amount of time students had been in contact with teachers. If students and teachers knew each other 0 or 1 week prior to the intervention, there was a significant expectancy effect. If they knew each other 2 or 3+ weeks the effect was not significant (although the precise cutoff might dependend on the scaling of weeks).  Effects of test type (individual or group) and test administrator awareness (“blind” or aware) were not significant and did not interact with length.

53.  Purpose-built  Comprehensive Meta-analysis (commercial)  Schwarzer (free, http://userpage.fu- berlin.de/~health/meta_e.htm)  Extensions to standard statistics packages  SPSS, Stata and SAS macros, downloadable from http://mason.gmu.edu/~dwilsonb/ma.html  Stata add-ons, downloadable from http://www.stata.com/support/faqs/stat/meta.html  HLM – V-known routine  MLwiN  MPlus

54.  Cooper, H., & Hedges, L. V. (Eds.) (1994). The handbook of research synthesis (pp. 521–529). New York: Russell Sage Foundation.  Hox, J. (2003). Applied multilevel analysis. Amsterdam: TT Publishers.  Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage Publications.  Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.

55.  Raudenbush, S.W. (1984). Magnitude of teacher expectancy effects on Pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.  Raudenbush, S.W. and Bryk, A.S. (2002). Hierarchical Linear Models (2 nd Ed.).Thousand Oaks: Sage Publications.  Download macros for free from http://mason.gmu.edu/~dwilsonb/ma.html http://mason.gmu.edu/~dwilsonb/ma.html  Download MLwiN for free from http://www.cmm.bristol.ac.uk/MLwiN/index.shtml http://www.cmm.bristol.ac.uk/MLwiN/index.shtml