1. Assessing Social-Emotional Skills at Scale:
Recent Developments in Policy, Practice, and Research
Martin R. West
Harvard Graduate School of Education
May 19, 2016

2. Policy Context: Required Accountability Indicators under ESSA
Academic achievement: proficiency rates (or alternative?) on state math/ELA tests
Another academic indicator:
High schools: graduation rates
Elementary/middle schools: academic growth (or another indicator of learning)
English language learners’ progress toward proficiency
At least one additional indicator of “school quality or student success”: Listed options include (but are not limited to) student engagement, educator engagement, access to and completion of advanced coursework, postsecondary readiness, or school climate and safety
Each indicator must receive “substantial” weight and the first three indicators together must receive “much greater” weight than the fourth

3. Requirements for “fifth indicator”
Policy Context:
Requirements for “fifth indicator”
Valid and reliable: draft regulations require that the indicator be predictive of achievement or graduation rates
Statewide: measured in a comparable way across all schools serving the same grade span
Disaggregated for schools and student subgroups
Meaningfully differentiate school performance: states need use the indicator to assign each school to one of at least three performance levels
Limited role (?): draft regulations state that the indicator can’t change identity of schools otherwise identified for comprehensive support

4. CORE Districts
A consortium of nine California school districts serving over one million students enrolled in more than 1,500 schools
Six of these districts have been operating since 2013 under an ESEA waiver to develop a new accountability system
School Quality Improvement Index (SQII) includes:
Academic domain: test score levels and growth; grad rates
Social-emotional domain: absences, suspensions, school climate and culture surveys, social-emotional skills
: field test of social emotional measures on ~450,000 students in grades 3-12
: plan to report publicly and include as 8% of SQII ratings

5. Potential concerns with use of self-report surveys for accountability purposes
Reference bias: students’ normative expectations may vary across schools, undermining comparability of responses (West et al., 2015)
Faking: because self-report surveys are notably easy to game, increasing risk of corruption due to stakes (Campbell, 1976)
Imprecision: ability to differentiate school performance when aggregated is unknown (Kane & Staiger, 2002)
Instructional response: superficial measures could lead to superficial pedagogy

6. Selection of SEL Measures
Three criteria: evidence that the skills were (1) measurable (in <20 minutes total), (2) meaningfully predictive of academic and life outcomes, and (3) malleable through school-based interventions

7. The measures were piloted, field tested, and rolled out at scale over the course of three years
~9000 students
participate in pilot of
survey-based
social-emotional measures
450,000+ students
participate in field test of survey-based social-emotional measures
All CORE schools
administer survey-based social-emotional measures as
part of the SQII

8. Internal reliability: full sample
Cronbach's alpha is a measure of internal consistency, or, how closely related a set of items are as a group. It is considered to be a measure of scale reliability.
Rule of thumb: 0.8 is good and 0.7 is acceptable

9. Internal reliability: key subgroups

10. Relationship between combined SEL measure and math test scores, CORE middle schools

11. Student-level correlations of SEL measures and math test scores: overall and within-school, CORE middle schools
Reference bias could make comparisons of their responses across schools misleading
we compared the strength of the student-level correlations between social-emotional skills and academic indicators overall (i.e., across all students attending CORE middle schools) with those obtained when we limit the analysis to comparisons of students attending the same school.
If students in higher-performing schools rate themselves more critically, then average self-ratings in those schools will be artificially low. This would cause the overall correlation to be biased downward, and lower than that observed among students responding to surveys within the same school environment.
We do not observe this pattern

13. Tentative lessons from CORE
CORE Field Test presents a unique opportunity to learn about the properties of self-report measures of social-emotional skills when administered at scale
Evidence is generally encouraging with respect to reliability (above grade 4) and validity both within and across schools
Ability to differentiate schools with certainty based on average performance is limited, but comparable to that of test scores
Key caveats: Field Test data cannot address…
how the performance of self-report measures would change if stakes were attached; and
how the reporting of measures will change instructional practice and, ultimately, student achievement and life outcomes.

14. Student perception of school climate surveys
Key choice (1): assess status of social-emotional skills directly or the conditions we believe foster them?
School conditions
Student perception of school climate surveys
Resources
Student status
Self reports
Teacher reports
Performance tasks
Key challenge: lack of knowledge of which conditions matter most
Key challenge: lack of information on skills driving any differences in performance

15. Key choice (2): where in the pipeline should we focus measurement activity?
School conditions
Student status
Proximal student outcomes
Intermediate student outcomes
Long-term student outcomes
Indicators
Curriculum & instruction
School climate
Non-academic resources
Academic knowledge & skills
Social-emotional skills
Course performance
Behavior
Academic achievement
School engagement
Educational attainment
Labor market success
Health, happiness, etc.
Measures
Student perception surveys
Resource inventories
Inspections
Classroom tests
Self-report surveys
Teacher-report surveys
Performance tasks
Course grades
Attendance
Merits and demerits
Office referrals
State tests & high school graduation
Chronic absenteeism
Suspensions & expulsions
Post-secondary success
Earnings & employment
???

16. Alternative uses of measures of social-emotional learning and/or school climate in ESSA
Accountability: the fifth indicator
Transparency: element in school report cards/dashboards
Diagnosis: focus of required “needs assessment” for schools identified for comprehensive/targeted support and improvement
Research & Development: include in state longitudinal data system to facilitate research