Construct Validity of Classroom Observations: Items, Factors, Raters, and Achievement Lee Branum-Martin, Coleen D. Carlson, Angelia Durand, Christopher Barr Texas Institute for Measurement, Evaluation, and Statistics University of Houston Society for Research on Educational Effectiveness March 4, 2010

= items + construct + rater + time + school + grade... Classroom Quality A Generalizability Theory Approach (following Raudenbush) Generalizability Raudenbush (2008). Statistical inference when classroom causality is measured with error. Context measurement Raudenbush & Sampson (1999). Ecometrics. Raudenbush, Rowan, & Kang (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data.

An Ecometric Approach (in response to Raudenbush) Development Conversation Uses Vocabulary Oral Language Item score = construct + rater + time + school + grade... Furnishings Arrangement Engagement Organization

Rater Differences Development Conversation Uses Vocabulary Oral Language Furnishings Arrangement Engagement Organization Rater Ratings are valid, but differ in severity (factor means) Ratings are valid, but differ in factor variances Ψ 11 Ψ 22 Ψ 21 Raters differ in validity λ 11 λ 12 λ 13 λ 14 λ 25 λ 26 λ 27 Item score = construct + rater + time + school + grade...

Time Differences Development Conversation Uses Vocabulary Oral Language Furnishings Arrangement Engagement Organization Time Classroom factors differ over time The variances & relations among classroom factors differ over time Ψ 11 Ψ 22 Ψ 21 Factor composition differs over time: the nature of classroom ecology changes λ 11 λ 12 λ 13 λ 14 λ 25 λ 26 λ 27 (School year, semester, month, class session, segment of session) Item score = construct + rater + time + school + grade...

Sample & Design Grade Unique YearSemesterK123Total TeachersSchools Fall Spring Fall Spring Fall Spring Total1,0061,0431, ,0152,172158

Example Items for Classroom Management Understanding of rules 1.Children appear to have limited understanding of rules and routines. This is evident in the classroom as children engage frequently in conflicts and rarely in purposeful activity. 2.Children appear to understand regular rules and routines, but there is occasionally a need for teacher reminders or reinforcement about some rules and routines. 3.Children appear to have internalized regular rules and routines. This is evident as children move through the classroom period smoothly, with few conflicts, and are most often seen engaged in purposeful activity. Communication of expectations 1.Expectations for children's behavior may be confusing or inconsistent, conflicts may be inconsistently resolved. 2.Expectations for children's behavior are communicated from teacher to children. 3.Clear expectations for children's behavior are consistently communicated in multiple ways. adapted from the ELLCO (Smith et al., 2002)

Fit Statistics: Grades, Years, Semesters StepModel  dfCFITLIRMSEAWRMR Fall Spring Fall Spring Fall * Spring Fall across years, grades Spring across years, grades Scoring all waves

Model Results: Factor Loadings & Thresholds Organization Reading Instruction Writing Instruction Factor Item Factor z-score Threshold between high and medium quality Threshold between medium and low quality Factor loading

Model Results: Factor Loadings & Thresholds Assessment Curriculum Oral Language Management Climate FactorItem — Factor z-score medium/high low/medium

Factor Correlation Matrix FactorAssessClimateCurric.Man.OralOrg.Read.Writ. Assessment1.00 Climate Curriculum Management Oral Language Organization Reading Writing

Interobserver Agreement on Factor Scores ObserverComparisonpTeacherObserver FactorrmeanSDmeanSD(diff)ICC Assessment Climate Management Oral Language Organization Reading Writing Interobserver correlation Mean differencesCross-classified

Correlations to campus mean achievement Reading Instruction Oral Language Instruction Classroom Management TestGradeYearSchoolsFallSpringFallSpringFallSpring ITBS TAKS * * ** ** * *** *** * * * * * * ** * p <.05

Conclusions Confirmatory factor models can be applied to observational data to examine relations among items, constructs, raters, time, and sites CFA on categorical items reveals functional information about the items. CFA incorporates theory and design in a falsifiable way. CFA is complex, but can serve as a first-stage validity analysis to be exported into other analyses, such as G- theory or multilevel models of outcomes.

Questions, Comments? References Raudenbush, S. W. (2008). Statistical inference when classroom causality is measured with error. Paper presented at the SREE. Raudenbush, S. W., Rowan, B., & Kang, S. J. (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data. J Ed Stat, 16, Raudenbush, S. W., & Sampson, R. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Soc Methodology, 29(1), Smith, M. W., Dickinson, D. K., Sangeorge, A., & Anastasopoulos, L. (2002). Early Language & Literacy Classroom Observation (ELLCO) Toolkit, Research Edition. Baltimore, MD: Paul H. Brookes.