1. Construct Validity of Classroom Observations: Items, Factors, Raters, and Achievement Lee Branum-Martin, Coleen D. Carlson, Angelia Durand, Christopher Barr Texas Institute for Measurement, Evaluation, and Statistics University of Houston www.times.uh.edu Society for Research on Educational Effectiveness March 4, 2010

2. = items + construct + rater + time + school + grade... Classroom Quality A Generalizability Theory Approach (following Raudenbush) Generalizability Raudenbush (2008). Statistical inference when classroom causality is measured with error. Context measurement Raudenbush & Sampson (1999). Ecometrics. Raudenbush, Rowan, & Kang (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data.

3. An Ecometric Approach (in response to Raudenbush) Development Conversation Uses Vocabulary Oral Language Item score = construct + rater + time + school + grade... Furnishings Arrangement Engagement Organization

4. Rater Differences Development Conversation Uses Vocabulary Oral Language Furnishings Arrangement Engagement Organization Rater Ratings are valid, but differ in severity (factor means) Ratings are valid, but differ in factor variances Ψ 11 Ψ 22 Ψ 21 Raters differ in validity λ 11 λ 12 λ 13 λ 14 λ 25 λ 26 λ 27 Item score = construct + rater + time + school + grade...

5. Time Differences Development Conversation Uses Vocabulary Oral Language Furnishings Arrangement Engagement Organization Time Classroom factors differ over time The variances & relations among classroom factors differ over time Ψ 11 Ψ 22 Ψ 21 Factor composition differs over time: the nature of classroom ecology changes λ 11 λ 12 λ 13 λ 14 λ 25 λ 26 λ 27 (School year, semester, month, class session, segment of session) Item score = construct + rater + time + school + grade...

6. Sample & Design Grade Unique YearSemesterK123Total TeachersSchools 06-07 Fall219216210205850 117 Spring197207199203806800106 07-08 Fall198205196185784776102 Spring193214202191800767102 08-09 Fall1021011029540037850 Spring971009484375 50 Total1,0061,0431,0039634,0152,172158

7. Example Items for Classroom Management Understanding of rules 1.Children appear to have limited understanding of rules and routines. This is evident in the classroom as children engage frequently in conflicts and rarely in purposeful activity. 2.Children appear to understand regular rules and routines, but there is occasionally a need for teacher reminders or reinforcement about some rules and routines. 3.Children appear to have internalized regular rules and routines. This is evident as children move through the classroom period smoothly, with few conflicts, and are most often seen engaged in purposeful activity. Communication of expectations 1.Expectations for children's behavior may be confusing or inconsistent, conflicts may be inconsistently resolved. 2.Expectations for children's behavior are communicated from teacher to children. 3.Clear expectations for children's behavior are consistently communicated in multiple ways. adapted from the ELLCO (Smith et al., 2002)

8. Fit Statistics: Grades, Years, Semesters StepModel  dfCFITLIRMSEAWRMR 206-07 Fall317.889.958.980.0551.13 06-07 Spring290.089.959.980.0531.09 07-08 Fall434.284.925.955.0731.42 07-08 Spring363.584.937.960.0661.31 08-09 Fall * 263.164.926.957.0911.21 08-09 Spring202.259.952.968.0801.11 3Fall across years, grades768.894.943.971.0601.71 Spring across years, grades661.893.951.973.0561.62 4Scoring all waves849.477.968.979.0502.32

9. Model Results: Factor Loadings & Thresholds Organization Reading Instruction Writing Instruction.62.84.90.70.77.86.85.70.77 Factor Item Factor z-score Threshold between high and medium quality Threshold between medium and low quality Factor loading

10. Model Results: Factor Loadings & Thresholds Assessment Curriculum Oral Language Management Climate FactorItem.85.90.94.48.84.88.77 1.00.77.92 — Factor z-score medium/high low/medium

11. Factor Correlation Matrix FactorAssessClimateCurric.Man.OralOrg.Read.Writ. Assessment1.00 Climate.531.00 Curriculum.58.551.00 Management.64.56.621.00 Oral Language.61.77.63 1.00 Organization.63.56.59.791.00 Reading.45.48.39.53.49.521.00 Writing.37.49.55.57.27.37.401.00

12. Interobserver Agreement on Factor Scores ObserverComparisonpTeacherObserver FactorrmeanSDmeanSD(diff)ICC Assessment.76.02.65-.06.76.43.13.23 Climate.81-.09.75-.15.78.33.21.19 Management.73-.08.56-.09.57.99.24.19 Oral Language.84.01.42-.05.47.11.19.20 Organization.49-.08.38-.02.34.19.14.25 Reading.86-.01.59-.08.67.30.17.18 Writing.70-.09.51-.17.54.21.14.20 Interobserver correlation Mean differencesCross-classified

13. Correlations to campus mean achievement Reading Instruction Oral Language Instruction Classroom Management TestGradeYearSchoolsFallSpringFallSpringFallSpring ITBS106-0778.21.38.20.39.24.38 07-0868.12-.05.17-.05.23.03 08-0933.01.14-.06.10-.16.09 206-0778.06.37.13.36.26.31 07-0869.12-.01.16.06.33.11 08-0933.39.57.39.56.22.54 TAKS306-07105.22.21.26.18.31 07-0896.13.19.08.16.15.22 08-0946.01.20.03.11.14.33 * * ** ** * *** *** * * * * * * ** * p <.05

14. Conclusions Confirmatory factor models can be applied to observational data to examine relations among items, constructs, raters, time, and sites CFA on categorical items reveals functional information about the items. CFA incorporates theory and design in a falsifiable way. CFA is complex, but can serve as a first-stage validity analysis to be exported into other analyses, such as G- theory or multilevel models of outcomes.

15. Questions, Comments? Lee.Branum-Martin@times.uh.edu References Raudenbush, S. W. (2008). Statistical inference when classroom causality is measured with error. Paper presented at the SREE. Raudenbush, S. W., Rowan, B., & Kang, S. J. (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data. J Ed Stat, 16, 295-330. Raudenbush, S. W., & Sampson, R. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Soc Methodology, 29(1), 1-41. Smith, M. W., Dickinson, D. K., Sangeorge, A., & Anastasopoulos, L. (2002). Early Language & Literacy Classroom Observation (ELLCO) Toolkit, Research Edition. Baltimore, MD: Paul H. Brookes.