1. Inferences about School Quality using opportunity to learn data: The effect of ignoring classrooms. Felipe Martinez CRESST/UCLA CCSSO Large Scale Assessment Conference Boston, MA; June 21, 2004

2. Introduction We focus on two related issues concerning the valid use of measures of school performance in accountability systems: differences in achievement (and OTL) between classrooms, and the impact on measures of school quality of ignoring the classroom context. Overview of research questions and studies. 1. Comparison of teacher and student reports of OTL 2. Distribution and effects of OTL (as reported by students and teachers) on student achievement in Reading. 3. Effect of ignoring classroom nesting in multilevel models. Effect on model-based measures of school quality.

3. Research Questions Comparison of teacher perceptions of the OTL they provide, to student perceptions of the OTL they receive. Factor Analysis Distribution and effects of OTL, as perceived by teachers and students. Multilevel Models (HLM) Distribution of student achievement and effect of ignoring classroom nesting on measures of school quality. Multilevel Models (HLM, Empirical Bayes estimates)

4. Factor Analysis Sample: Our sample consisted of 97,675 students attending elementary schools in grades 2 nd to 5 th. In addition the sample included data from 6,902 teachers. Methods: A questionnaire was used to collect OTL data from students and their teachers in order to compare their perceptions of the educational activities that occur in the classroom during the school year. The Student and Teacher OTL questionnaires were identical and consist of seven four-point Likert items related to content exposure and other classroom practices theoretically related to student performance in Language Arts. Each item inquires about the frequency with which the students performed a certain activity in the classroom during the school year. The scale ranges from 1 (almost never) to 4 (almost every day).

5. Factor Analysis Teacher and Student reports of Opportunity to Learn

6. Factor Analysis Teacher and Student reports of Opportunity to Learn

7. Factor Analysis Teacher and Student reports of Opportunity to Learn Differences exist in the way teachers and their students perceive the nature of teaching and learning activities conducted in the classroom As Teachers, Students clearly separated activities related to writing; however, students regarded activities led by the teacher, independently of whether these involved reading aloud to them or explaining grading criteria, as part of a single construct of Teacher activities. This may reflect a certain degree of confusion in the students (or lack of direction from the teachers) in terms of the specific nature of the activity being carried out by the teacher — whereby students may for example perceive explanation of grading criteria as their teacher simply reading something to them.

8. Multilevel Analysis Sample: The sample for multilevel modeling includes 46,284 2 nd to 5 th grade students distributed across 4,972 classrooms (teachers), in 375 elementary schools Methods: For each student achievement scores (SAT9 Reading) and background information was available, as well as context data about classrooms and schools. The factors created in the previous step are used as indicators of student and teacher OTL. At this stage, three-level multilevel models (HLMs) were employed to correctly take into account the nested structure of the data—students nested within classrooms, which in turn were nested in schools.

9. Multilevel Analysis Unconditional 2-Level Model

10. Multilevel Analysis Unconditional 3-Level Model

11. Multilevel Analysis Unconditional Two- and Three- Level Models of SAT9 Reading Scores

12. Multilevel Analysis Unconditional Three- Level Models of Student OTL

13. Multilevel Analysis Student- and Teacher- reported OTL as predictor of student achievement

14. Multilevel Analysis Student- and Teacher- reported OTL as predictor of student achievement

15. Multilevel Analysis Variation of Student-reported OTL effects across classrooms

16. Multilevel Analysis Correlations of EB School residuals from unconditional 2- and 3-level models

17. Multilevel Analysis Conditional two-level model with school context Level-1 (Student) Level-2 (School)

18. Multilevel Analysis Conditional three-level model with classroom context Level-1 (Student) Level-2 (Classroom)  00k =  000 + u00k Level-2 (School)

19. Multilevel Analysis Correlations of EB School residuals from conditional 2- and 3-level models (considering classroom context)

20. Discussion Teachers and students do not necessarily perceive the same opportunities to learn within a classroom. Results are in agreement with previous research (e.g. see Muthen et. al., 1995) suggesting that OTL information collected from teachers may not add significantly to what is known from information collected from students. Students’ own perceptions of OTL are more closely linked to their achievement than are the perceptions of teachers of the opportunities they provide the same students. Although OTL is provided to student at the classroom level, measuring student perceptions may be a more powerful (and accurate) indicator of OTL than teacher reports.

21. Discussion Results also support the notion that the classroom environment is at least as important or more important than the larger school as determinant of student learning (see Kyriakydes, Campbell & Gagatsis, 2000; Hill & Rowe, 1996; Anderson, 1987, among others). Student OTL slopes vary significantly across classrooms (Level-2), but not across schools (Level-3). This implies that the exact effects of OTL at any given classroom can differ considerably from the average. Model-based measures of school quality (in our case Empirical Bayes school-level residuals) are impacted by model choice. Estimates for particular schools can differ considerably depending on whether the classroom environment is included by using a three-level model, or not.

22. Discussion Results emphasize the importance of using three-level models. In general, d espite the fact that accountability systems are aimed at schools, results indicate that careful attention needs to be paid also to classroom differences within schools. Increasing attention to teacher effects is an encouraging sign although the use of these estimates in high stakes situations is problematic (McCaffrey et. al., 2004) Even within three-level models, however, use of EB residuals and other estimates of school quality for accountability purposes should be carefully considered as they constitute “at best, Type A effects” not suitable for accountability (Willms and Raudenbush, 1989; Raudenbush, 2004). Furthermore, the estimates of school performance we produce are cross-sectional in nature. For an up-to-date view on the promise but also the (sometimes overwhelming) complexity of the models needed for longitudinal studies of teacher and school effectiveness see the last issue of the Journal for Educational and Behavioral Statistics on Value Added Assessment (Spring 2004).