1 Testing 1, 2, 3: An Analysis of 4Sight in Pennsylvania A Paper Presented at the 2009 North American Summer Meeting of the Econometric Society Robert P. Strauss and Abby Clay Turner Carnegie Mellon University Saturday, June 6, 2009

2 Outline Overview of Testing in PA Schools Research Questions & Motivation Results: Alignment Analysis & Use of 4Sight Econometric Design & Results: –Program Evaluation –Reporting Category Analysis Conclusions and Policy Implications

3 Pennsylvania System of School Assessment (PSSA) 2002 No Child Left Behind (NCLB) Requires state testing to receive federal funds All students must be tested/reported in math/reading States set Adequate Yearly Progress (AYP) goals for their schools/districts –100% of students proficient by 2014 –Failing AYP leads to sanctions PSSA becomes high-stakes for schools

4 Overview of 4Sight Example of an intervention schools take to help meet AYP Math and reading tests written by Success for All Foundation of Maryland “Predictive Benchmark Assessment” –Given to students 4-5 times throughout the year –Meant to predict student performance on PSSA –Provides teachers with feedback on student strengths/weaknesses Schools buy in: –$1000 online, about $3/student/subject paper –Teacher/administrator training –4-5 hours of teaching time for tests alone –Teacher time to adapt to feedback : Used by 750 schools/310 districts

5 PSSA Design Multiple choice questions drawn from a set of “eligible content” for each grade/subject. –Aligned to the set of knowledge districts teach within the grade/subject Material within a grade/subject broken down: –Reporting Category Assessment Anchor –Eligible Content Math Reporting Categories: Numbers and Operations, Measurement, Geometry, Algebraic Concepts, Data Analysis and Probability Reading Reporting Categories: Comprehension and Reading Skills, Interpretation and Analysis of Fictional and Nonfictional Text

6 Research Question & Motivation How does 4Sight impact student PSSA performance? –Analyze use and alignment of 4Sight ( ) –Perform program evaluation Policy Motivations: –Measure benefits and compare to costs –Suggest improvements to process Economic Motivations: –Contribute to empirical education production literature

7 Analysis of 4Sight: Alignment 4-5 tests per subject/grade questions per test Same “eligible content” covered on each test –Does not cover all PSSA eligible content –6 th grade test covers only 40% of reading Only 80% of math content Exam not fully aligned to curriculum & PSSA

8 Analysis of 4Sight: Use Formative assessment: “Ongoing assessment designed to make student thinking visible to both teachers and students.” (National Research Council) Training materials provide feedback to teachers Students do not receive direct feedback 4Sight does not conform to NRC’s definition of formative assessment

9 How will using 4Sight affect PSSA? Positive effect if: –4Sight provides effective feedback to teachers Teachers use feedback to tailor their teaching –Students become accustomed to the format of 4Sight/PSSA with practice Negative effect if: –Teaching time is less (equivalent to 8 hours of instruction if using both tests) –Students are frustrated by being tested on material they have not yet learned –Students develop negative attitude toward test

10 Difficulty Identifying the Effect of 4Sight Interested in knowing change in test scores Evaluating “treatment effect” –Never observe treated and non-treated scores from same student –Selection bias: Are non-treated students appropriate controls for treated students? Modeling education production function –Student “ability” is unobserved –Data limitations to what we can control for

11 Data Student-level test score data plus identifying, SES, and academic (DRC) –Raw score, scaled score, “reporting category” –Matched with for 5 th to 6 th grade: 89.0% School-level data (PDE) 4Sight use by school, grade, subject (PDE) Descriptive statistics show students using 4Sight –4Sight users have higher income, higher % white, more experienced teachers –Still more likely to receive other treatments (Tutoring, FRL, Title III) –Begin with lower PSSA scores

12 Individual-Level OLS Equivalent to a 1.7-point increase at the mean Equivalent to a 2.4-point decrease at the mean

13 Individual-Level Quantile Regressions Reject the hypothesis that the coefficient on 4Sight is different at different quantiles for Reading specifications (3) & (4)

14 School-Level OLS

15 Propensity Score Matching Probit specification Matching: –School-level nearest neighbor based on propensity score –Trimmed to common support –Bootstrapped standard errors

16 Individual-Level Reporting Category Tobit Analysis: Math

17 Individual-Level Reporting Category Tobit Analysis: Reading

18 Conclusions/Policy Implications 4Sight not fully aligned with PA standards –Suggest rewriting test to full alignment –Possible within current length of test –Correlation between coverage and performance 4Sight does not conform with NRC’s definition of formative assessment –Provide format for formal feedback to teachers and students to realize full potential as a teaching tool Study of first year suggests small impact –Reading negative and significant in all specifications –Differences between treated and non-treated groups suggest possible endogeneity –Careful empirical analysis to continue for several years to determine the effects of 4Sight