ELM DICIPE Mozambique Gaza, Nampula, and Tete Midline 2016

Background Research Design Treatment & Comparison groups not assigned randomly Originally hoped to be true baseline/endline study, but insufficient identifiers at baseline made it impossible to reliably interview the same children at endline Some issues with sampling and potential bias: children in comparison areas were selected at random from the community whereas children in treatment areas were selected at random from ECD centers While we attempt a number of analytical approaches to minimize bias, strong conclusions about program impact cannot be made based on these analyses DICIPE 2016: Endline Review 28 December 2016

Analyses & Limitations Background Analyses & Limitations Analyses Four analyses presented in this report Balance test of background characteristics at baseline and endline to assess comparability Comparison of Comparison and Treatment adjusted mean scores at endline Comparison mean scores from baseline to endline Pseudo-gain scores analysis Limitations Scoring problems on some IDELA items raises questions about fidelity of baseline data Differences in sampling strategy in Treatment and Comparison areas limits comparability of full sample and may bias comparisons Systematic differences between children in Treatment and Comparison areas Unable to match majority of children at baseline to endline make gain scores suspect unless missingness was total random DICIPE 2016: Endline Review 28 December 2016

Background IDELA Instrument Motor Development Fine and gross motor skills: Hopping; Copying shape; Folding paper; Drawing Emergent Language and Literacy Print Awareness; Expressive & Receptive Language; Letters; Phonological Awareness Emergent Math/ Numeracy Number Sense; Shapes & Spatial Relations; Sorting; Problem Solving; Measurement & Comparison Social-Emotional Development Empathy; Emotional awareness; Self awareness; Solving conflict; Peer relationships Learning Approaches Executive Function International Development and Early Learning Assessment Internationally tested and locally adapted measure of fundamental skills children require to be ready for school. DICIPE 2016: Endline Review 28 December 2016

Background Sample Full sample comprises 1,557 observations Province Slightly more observations at endline than baseline Subsequent analyses use subsets of this data (data with missing variables is omitted for regression analysis) Province Baseline Endline Total Gaza 225 247 472 Nampula 220 302 522 Tete 267 296 567 712 845 1,557 Status Baseline Endline Total Comparison 349 363 712 Treatment 368 477 845 717 840 1,557 DICIPE 2016: Endline Review 28 December 2016

Balance test Explanation Purpose Analytical Approach Takeaway Assess the comparability of treatment and comparison groups at baseline and endline. Analytical Approach Compare mean values of background characteristics at baseline and endline, using t- tests with clustered standard errors to determine if differences are significant. Takeaway If treatment and comparison groups are similar on background characteristics, it makes the argument that differences in outcomes between groups represent the impact of the program stronger. Significant differences between groups limit the strength of subsequent analyses. DICIPE 2016: Endline Review 28 December 2016

Significant Difference? Balance test Baseline differences by treatment group (select variables) Variable Comparison Mean Treatment Mean Significant Difference? Child’s age 3.52 3.57 Mother is Literate 52% 53% Father is Literate 66% 76% ~ Number of Children in Family 4.04 3.95 Standardized SES Score 0.01 0.11 Number of Possessions 3.38 3.71 Number of Toy Types 2.32 2.40 Family owns a storybook 28% 30% Number of Reading Materials 1.20 1.21 Sum of Positive Learning Activities 2.46 2.80 Parent hits child 46% Sum of Negative Discipline 1.66 1.48 Composite Home Learning Attitudes Score 0.80 1.33 * DICIPE 2016: Endline Review 28 December 2016

Significant Difference? Balance test Endline differences by treatment group (select variables) Variable Comparison Mean Treatment Mean Significant Difference? Child’s age 3.68 4.10 *** Mother is Literate 36% 46% Father is Literate 58% 69% Number of Children in Family 3.89 4.19 * Standardized SES Score -0.28 0.12 ** Number of Possessions 3.13 3.82 Number of Toy Types 2.85 3.50 Family owns a storybook 18% 28% Number of Reading Materials Sum of Positive Learning Activities 1.65 2.13 Parent hits child 41% 38% Sum of Negative Discipline 1.43 1.36 Composite Home Learning Attitudes Score 1.71 2.28 ~ DICIPE 2016: Endline Review 28 December 2016

Balance test Summary At baseline, Comparison & Treatment groups appear relatively similar Relatively few significant differences between children in background/family characteristics at baseline Children in treatment areas came from families with slightly more literate fathers and slightly better home learning environments Appropriate to compare developmental outcomes among these groups At endline, Comparison & Treatment groups are quite different Children from treatment areas at endline are older, come from families of higher SES and have more possessions (these factors are unlikely to have been influenced by the program) Children from treatment areas at endline come from significantly better home learning environments, have more books at home, and benefit from more supportive learning activities with their parents (these factors may be have been influenced by the program) Comparing outcomes between these groups comes with substantial limitations. DICIPE 2016: Endline Review 28 December 2016

Endline results Explanation Purpose Analytical Approach Takeaway Examine item-by-item and domain averages and differences between treatment and comparison groups at endline. Assess strong and weak areas for children. Analytical Approach “Fitted mean” values for treatment and comparison children are presented after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences. Takeaway Endline differences between treatment and control should not be interpreted as program impact. While we control for many factors, the results of our balance tests suggest there were large and significant differences between treatment and control children that may be expressed in unobserved variables. These differences may bias our findings. DICIPE 2016: Endline Review 28 December 2016

Endline results Early Math Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001 DICIPE 2016: Endline Review 28 December 2016

Social Emotional Development Endline results Social Emotional Development Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001 DICIPE 2016: Endline Review 28 December 2016

Endline results Early Literacy Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001 DICIPE 2016: Endline Review 28 December 2016

Endline results Motor Development Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001 DICIPE 2016: Endline Review 28 December 2016

Executive Function & Approaches to Learning Endline results Executive Function & Approaches to Learning Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001 DICIPE 2016: Endline Review 28 December 2016

Endline results Summary Children in Treatment areas exhibit better outcomes than those in Comparison areas Large and significant differences on nearly every subtask at endline Differences between areas exist even after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences Differential sampling strategy may bias results Sampling strategy interviewed only those children in ECD centers in treatment areas whereas Comparison areas—these samples may not be representative of the overall population in these villages Inappropriate to interpret these differences as estimates of program impact DICIPE 2016: Endline Review 28 December 2016

Comparison of means Explanation Purpose Analytical Approach Takeaway Examine domain averages at baseline and endline and the differences between treatment and comparison groups and provinces. Estimate directionality of changes and make comparisons between provinces. Analytical Approach “Fitted mean” values are presented after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences. We compare baseline to endline, treatment to control, and the results from different provinces Takeaway Differences between baseline and endline should not be interpreted as true change over time. The endline sample comprises mostly different children than the baseline sample. These should be interpreted as independent snapshots of the communities at baseline and endline. DICIPE 2016: Endline Review 28 December 2016

Domains by baseline/endline and treatment status Comparison of means Domains by baseline/endline and treatment status Note: Significant differences between baseline and endline mean scores are noted independently for treatment and comparison areas (* = p < 0.05; ** = p < 0.01; *** = p < 0.001) DICIPE 2016: Endline Review 28 December 2016

Domains by baseline/endline and treatment status Comparison of means Domains by baseline/endline and treatment status Baseline scores No significant differences between treatment and control areas Endline scores Comparison areas had significantly lower endline scores (from baseline) in every domain except for Motor Development, Treatment areas had significantly lower endline scores (from baseline) in Early Math, Early Literacy, and Social Emotional Development. Treatment areas had significantly higher Motor Development at endline. Early Math and Total IDELA were unchanged from baseline. DICIPE 2016: Endline Review 28 December 2016

Comparison of means Endline by province Note: Significant differences are noted with Gaza as the reference province (* = p < 0.05; ** = p < 0.01; *** = p < 0.001) DICIPE 2016: Endline Review 28 December 2016

Comparison of means Endline by province Findings Children in Nampula scored significantly lower on every domain compared to children in Gaza Children in Tete scored significantly lower on Motor Development, Early Literacy, Social Emotional Development than children in Gaza Children in Tete scored significantly higher on Early Math No difference in overall IDELA score between Tete and Gaza DICIPE 2016: Endline Review 28 December 2016

Comparison of means Summary Large and significant differences between baseline and endline and by treatment Questions about fidelity of data collection at baseline and whether it truly represents early learning and development, especially given the large negative differences in control areas between baseline and endline Treatment is associated with no differences at baseline, but significant (and positive) differences at endline. These findings are positive, but the quality of the data and research design make strong conclusions impossibl Large and significant differences at endline scores between provinces Children in Nampula consistently performed lower than in other provinces Children in Gaza generally performed highest DICIPE 2016: Endline Review 28 December 2016

Pseudo gain-scores Explanation Purpose Analytical Approach Takeaway Examine a subset of children from Tete for whom we have both baseline and endline data and assess how they changed over time. Analytical Approach Endline data collection in Tete province appears to have been conducted with many of the same children that were interviewed at baseline. 147 children (68 comparison and 77 treatment) were able to be matched by name from baseline. This section calculates a gain score for each child (endline – baseline) and then evaluates whether treatment was associated with change over time. Takeaway Data at two points in time allows us to build a more convincing story of what changed over time and how it differed in intervention vs. comparison areas. While differences cannot be solely attributed to the treatment, by controlling for baseline scores, changes made over time provide more convincing evidence of a treatment effect. DICIPE 2016: Endline Review 28 December 2016

IDELA baseline & gain scores by treatment Pseudo gain-scores IDELA baseline & gain scores by treatment Note: Significant differences are noted between treatment and control at endline and baseline (* = p < 0.05; ** = p < 0.01; *** = p < 0.001) DICIPE 2016: Endline Review 28 December 2016

IDELA baseline & gain scores by treatment Pseudo gain-scores IDELA baseline & gain scores by treatment Findings Children in Tete surveyed at both endline and baseline demonstrated increases in early learning and development regardless of treatment status In general, children in intervention areas began with higher levels of development, with significantly higher Early Literacy and Early Math levels. Children in treatment areas had significantly larger gains in the domains of Social Emotional Development, Motor Development, and the Total IDELA score There were no differences in the average gains exhibited by children in Early Literacy or Early Math DICIPE 2016: Endline Review 28 December 2016

Conclusion Summary of findings Background characteristics of Treatment and Comparison groups were more different at endline than baseline Fewer significant differences at baseline than endline Groups are more comparable at baseline than endline Children in Treatment areas had higher scores at endline Cannot be interpreted as program impact Children exhibited similar scores at baseline Large and significant differences between provinces at endline Children in Nampula scored significantly lower than in Tete and Gaza In Tete, children in Treatment areas had larger gains Concentrated in Social Emotional Development and Motor Development domains Suggestive of an impact of programing, but not conclusive DICIPE 2016: Endline Review 28 December 2016

Conclusion Next steps Investigate reasons for differences between provinces Follow the same children next year for true gain-score analysis Strengthen estimates of program impact with supplemental analyses DICIPE 2016: Endline Review 28 December 2016