2. ELM DICIPE Mozambique Gaza, Nampula, and Tete Midline 2016

3. Background Research Design
Treatment & Comparison groups not assigned randomly
Originally hoped to be true baseline/endline study, but insufficient identifiers at baseline made it impossible to reliably interview the same children at endline
Some issues with sampling and potential bias: children in comparison areas were selected at random from the community whereas children in treatment areas were selected at random from ECD centers
While we attempt a number of analytical approaches to minimize bias, strong conclusions about program impact cannot be made based on these analyses
DICIPE 2016: Endline Review
28 December 2016

4. Analyses & Limitations
Background
Analyses & Limitations
Analyses
Four analyses presented in this report
Balance test of background characteristics at baseline and endline to assess comparability
Comparison of Comparison and Treatment adjusted mean scores at endline
Comparison mean scores from baseline to endline
Pseudo-gain scores analysis
Limitations
Scoring problems on some IDELA items raises questions about fidelity of baseline data
Differences in sampling strategy in Treatment and Comparison areas limits comparability of full sample and may bias comparisons
Systematic differences between children in Treatment and Comparison areas
Unable to match majority of children at baseline to endline make gain scores suspect unless missingness was total random
DICIPE 2016: Endline Review
28 December 2016

5. Background IDELA Instrument
Motor Development
Fine and gross motor skills: Hopping; Copying shape; Folding paper; Drawing
Emergent Language and Literacy
Print Awareness; Expressive & Receptive Language; Letters; Phonological Awareness
Emergent Math/ Numeracy
Number Sense; Shapes & Spatial Relations; Sorting; Problem Solving; Measurement & Comparison
Social-Emotional Development
Empathy; Emotional awareness; Self awareness; Solving conflict; Peer relationships
Learning Approaches
Executive Function
International Development and Early Learning Assessment
Internationally tested and locally adapted measure of fundamental skills children require to be ready for school.
DICIPE 2016: Endline Review
28 December 2016

6. Background Sample Full sample comprises 1,557 observations Province
Slightly more observations at endline than baseline
Subsequent analyses use subsets of this data (data with missing variables is omitted for regression analysis)
Province
Baseline
Endline
Total
Gaza
225
247
472
Nampula
220
302
522
Tete
267
296
567
712
845
1,557
Status
Baseline
Endline
Total
Comparison
349
363
712
Treatment
368
477
845
717
840
1,557
DICIPE 2016: Endline Review
28 December 2016

7. Balance test Explanation Purpose Analytical Approach Takeaway
Assess the comparability of treatment and comparison groups at baseline and endline.
Analytical Approach
Compare mean values of background characteristics at baseline and endline, using t- tests with clustered standard errors to determine if differences are significant.
Takeaway
If treatment and comparison groups are similar on background characteristics, it makes the argument that differences in outcomes between groups represent the impact of the program stronger. Significant differences between groups limit the strength of subsequent analyses.
DICIPE 2016: Endline Review
28 December 2016

8. Significant Difference?
Balance test
Baseline differences by treatment group (select variables)
Variable
Comparison Mean
Treatment Mean
Significant Difference?
Child’s age
3.52
3.57
Mother is Literate
52%
53%
Father is Literate
66%
76% ~
Number of Children in Family
4.04
3.95
Standardized SES Score
0.01
0.11
Number of Possessions
3.38
3.71
Number of Toy Types
2.32
2.40
Family owns a storybook
28%
30%
Number of Reading Materials
1.20
1.21
Sum of Positive Learning Activities
2.46
2.80
Parent hits child
46%
Sum of Negative Discipline
1.66
1.48
Composite Home Learning Attitudes Score
0.80
1.33 *
DICIPE 2016: Endline Review
28 December 2016

9. Significant Difference?
Balance test
Endline differences by treatment group (select variables)
Variable
Comparison Mean
Treatment Mean
Significant Difference?
Child’s age
3.68
4.10
***
Mother is Literate
36%
46%
Father is Literate
58%
69%
Number of Children in Family
3.89
4.19 *
Standardized SES Score
-0.28
0.12 **
Number of Possessions
3.13
3.82
Number of Toy Types
2.85
3.50
Family owns a storybook
18%
28%
Number of Reading Materials
Sum of Positive Learning Activities
1.65
2.13
Parent hits child
41%
38%
Sum of Negative Discipline
1.43
1.36
Composite Home Learning Attitudes Score
1.71
2.28 ~
DICIPE 2016: Endline Review
28 December 2016

10. Balance test
Summary
At baseline, Comparison & Treatment groups appear relatively similar
Relatively few significant differences between children in background/family characteristics at baseline
Children in treatment areas came from families with slightly more literate fathers and slightly better home learning environments
Appropriate to compare developmental outcomes among these groups
At endline, Comparison & Treatment groups are quite different
Children from treatment areas at endline are older, come from families of higher SES and have more possessions (these factors are unlikely to have been influenced by the program)
Children from treatment areas at endline come from significantly better home learning environments, have more books at home, and benefit from more supportive learning activities with their parents (these factors may be have been influenced by the program)
Comparing outcomes between these groups comes with substantial limitations.
DICIPE 2016: Endline Review
28 December 2016

11. Endline results Explanation Purpose Analytical Approach Takeaway
Examine item-by-item and domain averages and differences between treatment and comparison groups at endline. Assess strong and weak areas for children.
Analytical Approach
“Fitted mean” values for treatment and comparison children are presented after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences.
Takeaway
Endline differences between treatment and control should not be interpreted as program impact. While we control for many factors, the results of our balance tests suggest there were large and significant differences between treatment and control children that may be expressed in unobserved variables. These differences may bias our findings.
DICIPE 2016: Endline Review
28 December 2016

12. Endline results Early Math
Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001
DICIPE 2016: Endline Review
28 December 2016

13. Social Emotional Development
Endline results
Social Emotional Development
Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001
DICIPE 2016: Endline Review
28 December 2016

14. Endline results Early Literacy
Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001
DICIPE 2016: Endline Review
28 December 2016

15. Endline results Motor Development
Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001
DICIPE 2016: Endline Review
28 December 2016

16. Executive Function & Approaches to Learning
Endline results
Executive Function & Approaches to Learning
Note: Significant differences are noted with * = p < 0.05; ** = p < 0.01; *** = p < 0.001
DICIPE 2016: Endline Review
28 December 2016

17. Endline results Summary
Children in Treatment areas exhibit better outcomes than those in Comparison areas
Large and significant differences on nearly every subtask at endline
Differences between areas exist even after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences
Differential sampling strategy may bias results
Sampling strategy interviewed only those children in ECD centers in treatment areas whereas Comparison areas—these samples may not be representative of the overall population in these villages
Inappropriate to interpret these differences as estimates of program impact
DICIPE 2016: Endline Review
28 December 2016

18. Comparison of means Explanation Purpose Analytical Approach Takeaway
Examine domain averages at baseline and endline and the differences between treatment and comparison groups and provinces. Estimate directionality of changes and make comparisons between provinces.
Analytical Approach
“Fitted mean” values are presented after controlling for the child’s age, province, and caregiver-reported Socio-Economic Status, Home Learning Environment, Home Learning Activities, and accounting for village-by-village differences. We compare baseline to endline, treatment to control, and the results from different provinces
Takeaway
Differences between baseline and endline should not be interpreted as true change over time. The endline sample comprises mostly different children than the baseline sample. These should be interpreted as independent snapshots of the communities at baseline and endline.
DICIPE 2016: Endline Review
28 December 2016

19. Domains by baseline/endline and treatment status
Comparison of means
Domains by baseline/endline and treatment status
Note: Significant differences between baseline and endline mean scores are noted independently for treatment and comparison areas (* = p < 0.05; ** = p < 0.01; *** = p < 0.001)
DICIPE 2016: Endline Review
28 December 2016

20. Domains by baseline/endline and treatment status
Comparison of means
Domains by baseline/endline and treatment status
Baseline scores
No significant differences between treatment and control areas
Endline scores
Comparison areas had significantly lower endline scores (from baseline) in every domain except for Motor Development,
Treatment areas had significantly lower endline scores (from baseline) in Early Math, Early Literacy, and Social Emotional Development. Treatment areas had significantly higher Motor Development at endline. Early Math and Total IDELA were unchanged from baseline.
DICIPE 2016: Endline Review
28 December 2016

21. Comparison of means Endline by province
Note: Significant differences are noted with Gaza as the reference province
(* = p < 0.05; ** = p < 0.01; *** = p < 0.001)
DICIPE 2016: Endline Review
28 December 2016

22. Comparison of means Endline by province Findings
Children in Nampula scored significantly lower on every domain compared to children in Gaza
Children in Tete scored significantly lower on Motor Development, Early Literacy, Social Emotional Development than children in Gaza
Children in Tete scored significantly higher on Early Math
No difference in overall IDELA score between Tete and Gaza
DICIPE 2016: Endline Review
28 December 2016

23. Comparison of means Summary
Large and significant differences between baseline and endline and by treatment
Questions about fidelity of data collection at baseline and whether it truly represents early learning and development, especially given the large negative differences in control areas between baseline and endline
Treatment is associated with no differences at baseline, but significant (and positive) differences at endline. These findings are positive, but the quality of the data and research design make strong conclusions impossibl
Large and significant differences at endline scores between provinces
Children in Nampula consistently performed lower than in other provinces
Children in Gaza generally performed highest
DICIPE 2016: Endline Review
28 December 2016

24. Pseudo gain-scores Explanation Purpose Analytical Approach Takeaway
Examine a subset of children from Tete for whom we have both baseline and endline data and assess how they changed over time.
Analytical Approach
Endline data collection in Tete province appears to have been conducted with many of the same children that were interviewed at baseline. 147 children (68 comparison and 77 treatment) were able to be matched by name from baseline. This section calculates a gain score for each child (endline – baseline) and then evaluates whether treatment was associated with change over time.
Takeaway
Data at two points in time allows us to build a more convincing story of what changed over time and how it differed in intervention vs. comparison areas. While differences cannot be solely attributed to the treatment, by controlling for baseline scores, changes made over time provide more convincing evidence of a treatment effect.
DICIPE 2016: Endline Review
28 December 2016

25. IDELA baseline & gain scores by treatment
Pseudo gain-scores
IDELA baseline & gain scores by treatment
Note: Significant differences are noted between treatment and control at endline and baseline
(* = p < 0.05; ** = p < 0.01; *** = p < 0.001)
DICIPE 2016: Endline Review
28 December 2016

26. IDELA baseline & gain scores by treatment
Pseudo gain-scores
IDELA baseline & gain scores by treatment
Findings
Children in Tete surveyed at both endline and baseline demonstrated increases in early learning and development regardless of treatment status
In general, children in intervention areas began with higher levels of development, with significantly higher Early Literacy and Early Math levels.
Children in treatment areas had significantly larger gains in the domains of Social Emotional Development, Motor Development, and the Total IDELA score
There were no differences in the average gains exhibited by children in Early Literacy or Early Math
DICIPE 2016: Endline Review
28 December 2016

27. Conclusion Summary of findings
Background characteristics of Treatment and Comparison groups were more different at endline than baseline
Fewer significant differences at baseline than endline
Groups are more comparable at baseline than endline
Children in Treatment areas had higher scores at endline
Cannot be interpreted as program impact
Children exhibited similar scores at baseline
Large and significant differences between provinces at endline
Children in Nampula scored significantly lower than in Tete and Gaza
In Tete, children in Treatment areas had larger gains
Concentrated in Social Emotional Development and Motor Development domains
Suggestive of an impact of programing, but not conclusive
DICIPE 2016: Endline Review
28 December 2016

28. Conclusion
Next steps
Investigate reasons for differences between provinces Follow the same children next year for true gain-score analysis Strengthen estimates of program impact with supplemental analyses
DICIPE 2016: Endline Review
28 December 2016