Assessing Social-Emotional Skills at Scale: Recent Developments in Policy, Practice, and Research Martin R. West Harvard Graduate School of Education May 19, 2016

Policy Context: Required Accountability Indicators under ESSA Academic achievement: proficiency rates (or alternative?) on state math/ELA tests Another academic indicator: High schools: graduation rates Elementary/middle schools: academic growth (or another indicator of learning) English language learners’ progress toward proficiency At least one additional indicator of “school quality or student success”: Listed options include (but are not limited to) student engagement, educator engagement, access to and completion of advanced coursework, postsecondary readiness, or school climate and safety Each indicator must receive “substantial” weight and the first three indicators together must receive “much greater” weight than the fourth

Requirements for “fifth indicator” Policy Context: Requirements for “fifth indicator” Valid and reliable: draft regulations require that the indicator be predictive of achievement or graduation rates Statewide: measured in a comparable way across all schools serving the same grade span Disaggregated for schools and student subgroups Meaningfully differentiate school performance: states need use the indicator to assign each school to one of at least three performance levels Limited role (?): draft regulations state that the indicator can’t change identity of schools otherwise identified for comprehensive support

CORE Districts A consortium of nine California school districts serving over one million students enrolled in more than 1,500 schools Six of these districts have been operating since 2013 under an ESEA waiver to develop a new accountability system School Quality Improvement Index (SQII) includes: Academic domain: test score levels and growth; grad rates Social-emotional domain: absences, suspensions, school climate and culture surveys, social-emotional skills 2014-15: field test of social emotional measures on ~450,000 students in grades 3-12 2015-16: plan to report publicly and include as 8% of SQII ratings

Potential concerns with use of self-report surveys for accountability purposes Reference bias: students’ normative expectations may vary across schools, undermining comparability of responses (West et al., 2015) Faking: because self-report surveys are notably easy to game, increasing risk of corruption due to stakes (Campbell, 1976) Imprecision: ability to differentiate school performance when aggregated is unknown (Kane & Staiger, 2002) Instructional response: superficial measures could lead to superficial pedagogy

Selection of SEL Measures Three criteria: evidence that the skills were (1) measurable (in <20 minutes total), (2) meaningfully predictive of academic and life outcomes, and (3) malleable through school-based interventions

The measures were piloted, field tested, and rolled out at scale over the course of three years 2013-14 2014-15 2015-16 ~9000 students participate in pilot of survey-based social-emotional measures 450,000+ students participate in field test of survey-based social-emotional measures All CORE schools administer survey-based social-emotional measures as part of the SQII

Internal reliability: full sample Cronbach's alpha is a measure of internal consistency, or, how closely related a set of items are as a group. It is considered to be a measure of scale reliability. Rule of thumb: 0.8 is good and 0.7 is acceptable

Internal reliability: key subgroups

Relationship between combined SEL measure and math test scores, CORE middle schools

Student-level correlations of SEL measures and math test scores: overall and within-school, CORE middle schools Reference bias could make comparisons of their responses across schools misleading we compared the strength of the student-level correlations between social-emotional skills and academic indicators overall (i.e., across all students attending CORE middle schools) with those obtained when we limit the analysis to comparisons of students attending the same school. If students in higher-performing schools rate themselves more critically, then average self-ratings in those schools will be artificially low. This would cause the overall correlation to be biased downward, and lower than that observed among students responding to surveys within the same school environment. We do not observe this pattern

Tentative lessons from CORE CORE Field Test presents a unique opportunity to learn about the properties of self-report measures of social-emotional skills when administered at scale Evidence is generally encouraging with respect to reliability (above grade 4) and validity both within and across schools Ability to differentiate schools with certainty based on average performance is limited, but comparable to that of test scores Key caveats: Field Test data cannot address… how the performance of self-report measures would change if stakes were attached; and how the reporting of measures will change instructional practice and, ultimately, student achievement and life outcomes.

Student perception of school climate surveys Key choice (1): assess status of social-emotional skills directly or the conditions we believe foster them? School conditions Student perception of school climate surveys Resources Student status Self reports Teacher reports Performance tasks Key challenge: lack of knowledge of which conditions matter most Key challenge: lack of information on skills driving any differences in performance

Key choice (2): where in the pipeline should we focus measurement activity? School conditions Student status Proximal student outcomes Intermediate student outcomes Long-term student outcomes Indicators Curriculum & instruction School climate Non-academic resources Academic knowledge & skills Social-emotional skills Course performance Behavior Academic achievement School engagement Educational attainment Labor market success Health, happiness, etc. Measures Student perception surveys Resource inventories Inspections Classroom tests Self-report surveys Teacher-report surveys Performance tasks Course grades Attendance Merits and demerits Office referrals State tests & high school graduation Chronic absenteeism Suspensions & expulsions Post-secondary success Earnings & employment ???

Alternative uses of measures of social-emotional learning and/or school climate in ESSA Accountability: the fifth indicator Transparency: element in school report cards/dashboards Diagnosis: focus of required “needs assessment” for schools identified for comprehensive/targeted support and improvement Research & Development: include in state longitudinal data system to facilitate research