1. Carina Omoeva, FHI 360 Wael Moussa, FHI 360
Integrating Equity into Assessment of Learning Outcomes: a Structured Approach to Equity Analysis
Collaborative partnership focused on strengthening data and research evidence for equity in education
Premised on the idea that in order to have better understanding of equity, we need to take responsibility for producing better, deeper data
Carina Omoeva, FHI 360
Wael Moussa, FHI 360

2. The Structured Approach to Equity Analysis
Developed by the Education Equity Research Initiative as a process for performing analysis of outcomes in education with a consistent eye towards equity
Consists of a series of questions formulated to examine an aspect of the data, and coded in a Stata tool
Designed to provide a common framework for analysis across equity categories
The final aim of the sequence is to not only capture disparities in outcomes but also identify the change in equity of the program/policy impact
One of the first steps taken by the initiative to develop a common framework to apply to existing and future data
Addresses the question – what do we need to start looking at, in order to begin examining equity in our programsCarina
Preview of the panel: Fhi 360 – framework, Save the Children – within school and between school analysis; RTI

3. Level of Analysis
Dimension of Analysis
Questions
Student
Descriptive analysis to examine variability across key equity dimensions
Individual
What is the overall shape of the distribution of outcomes?
Background
Are the lowest performing students (at the bottom decile/ quintile of the distribution) substantively different from the higher performing students by gender, socioeconomic status, ethnicity/race, language, urban/rural, geographic location, disability status?
School
Cluster (school, district, subnational unit)
What is the overall school-level distribution for outcomes? How large is the variability in outcomes between clusters (schools, districts, regions)?
What is the student composition, along gender, socioeconomic, ethnic, and other student subgroups, among low performing schools?
Impact evaluation with lens on equity
Program/Intervention
What is the effect of the program on outcomes of interest? is it homogeneous or heterogeneous across the different student subgroups? Does the program/ policy have an equity building effect?
Inputs
Resource and input allocation
Classroom
What are pupil-teacher ratios for different schools or clusters? How large is the disparity between high performing and low performing schools?
What are the average teacher characteristics in low performing schools versus high performing schools?
Resources
How are public resources/ program resources allocated between schools?
Other
Adding a time dimension

4. Overview of the sequence
Identify equity dimensions and population composition relevant to the project context
Examine disparities in outcomes between individual students as well as groups of students (equity groupings)
Students can be grouped into performance categories
Construct statistical profiles for each category along observable student characteristics (including equity group membership)
Generate statistics of performance metrics and analyze distributions at aggregated levels (school, district, etc.)
Estimate program impacts, overall and for each group
Test for heterogeneity of estimated impacts

5. What are the equity dimensions present in the sample?
As a first step, we perform a simple tabulation of equity dimensions – the social and demographic characteristics predictive of outcomes
Recommended dimensions include gender, ethnicity (proxied by language), SES, disability
A crosstab helps determine if subgroup sample sizes are adequate for group-level inference
Additional dimensions – Practical Recommendations

6. What is the overall distribution of outcomes?
The overall shape of the distribution provides a gauge of the overall disparity across individuals and clusters of learners
Distributional plots are useful to examine disparities between groups, but also within groups

7. Separate lowest performing students from rest of sample
Pre-treatment
We divide students into performance groups to construct statistical profiles for each
With reading assessment, levels are based on reading ability
Students are identified as non-readers, non-fluent readers, and fluent readers
Cutscores are determined using ORF and reading comprehension performance
However, can be identified as bottom 10%, or bottom 25% of learners
Post-treatment

8. Are the lowest performing learners different on equity profile?
Non-Reader
Non-Fluent Reader
Fluent Reader
Age
9.51
9.24*
9.07*
Corridor:
Classroom and Teacher Characteristics:
Region 1
0.25
0.35*
0.41*
Multi-Grade Classroom
0.02
0.01
Region 2
0.48
0.42*
0.38*
Length of Period
38.96
39.53
37.66
Region 3
0.28
0.23
0.21
Class Size
31.96
32.31
32.74
Equity Groups:
Teacher is Male
0.53
0.59*
0.64*
Female
0.42
0.51*
Teacher is Female
0.47
0.36*
Male
0.58
0.49*
Teacher - Secondary Education or Lower
0.69
0.65
0.6*
Home Language - National
0.82
0.84
Teacher - Postsecondary Education
0.31
0.35
0.4*
International
0.18
0.16
Teacher Experience
11.57
10.03*
10.69
High SES
0.56*
Teacher is Salaried
0.87
0.89
0.96*
Low SES
0.52
0.44*
Observations
474
688
211
* p < 0.10

9. What is the distribution of lowest performers across clusters (schools)?
Using performance categories, we can examine the share of nonreaders between and within schools
Having non-performers clustered within some schools vs. spread evenly across all is potentially relevant for implementation
Can be repeated for larger units (e.g. districts)
In this example, we look at baseline and endline; only 20 percent of schools have less than 10% non-readers
Pre-treatment
Post-treatment

10. What is the effect of the program on outcomes?
In case of a true random experiment, simple differences in means can be attributed to the program
From an equity perspective, we must disaggregate program effects by equity dimension to test for impact heterogeneity

11. Examining Program Impact by Equity Dimension
A more elegant method of estimating program impacts is via regression analysis
In the case of longitudinal student data with two time periods, we apply a difference-in-differences approach with multi-level effects (school and student)
In the case of true randomization of the treatment, HLM (random effects) would be the preferred model specification—otherwise, student-level fixed effects
To test for heterogeneity of impact, we interact the treatment*post variable with equity group indicators ( 𝑰 𝒈 ), as follows:
𝑌 𝑖𝑠𝑡 =𝛼𝑇𝑟𝑒𝑎𝑡+𝛽𝑃𝑜𝑠𝑡+∑ 𝛾 𝑔 𝐼 𝑔 +∑ 𝛿 𝑔 𝑇𝑟𝑒𝑎𝑡∗𝑃𝑜𝑠𝑡∗ 𝐼 𝑔 + 𝜐 𝑖 + 𝜐 𝑠 + 𝜀 𝑖𝑠𝑡
In this case, the impact parameters are 𝛿 1 , 𝛿 2 , …, 𝛿 𝑔

12. Program Impact by Equity Dimension
(1)
(2)
(3)
(4)
Program Impact:
Overall
+4.26***
Effect on Males
+3.76***
Effect on Females
+4.79***
Effect on National Home Language
+4.39***
Effect on International Home Language
+3.58**
Effect on High SESa
+5.29***
Effect on Low SESa
+2.97*
In this example, the overall program impact is estimated at WPM
Differences in coefficients are testable using simple t-tests available with most standard statistical software
Same principle applies to any other educational outcome
Note: *** p < .01, ** p < .05, * p < .10; a denotes statistically significant difference in impact estimates between groups

13. Does the program have an equity-building effect?

14. Does the program have an equity-building effect? – Another example
In this example, the program built equity on the gender dimension, but worsened it for the wealth dimension

15. Additional analyses
Examining resource inputs and financing across schools
Additional time dimensions to examine change in trajectory

16. Concluding Remarks
The proposed structured approach to equity analysis can be applied to different outcomes, and to both exploratory/ descriptive and impact studies
Stata tool provides standardized code that can be adapted to any dataset with individual-level data
The approach is also flexible with the number of equity dimensions and degree of fragmentation in terms of student population make up
Additional steps should include examining resource inputs and financing across schools, and added time dimensions to examine change in trajectory

17. THANK YOU LEARN MORE: www.educationequity2030.org FOLLOW US
@equity2030 | #equity2030
SUPPORT US