1. ELM and HEART Baseline report October 2015
Cindy Sobieski, Specialist, Research Learning
Lauren Pisani, Senior Specialist, Research Learning
Add others names?

2. Overview I. Research questions II. Sample III. Tools used
Motor Development
Emergent Numeracy
Emergent Literacy
Socio-emotional Development
SEARS-PRE
IV. Baseline skill scores
V. Conclusion

3. I. Research questions
Key question for the baseline: Are Malawian children in both Heart & ELM, ELM only, HEART only and Control similar in terms of their measurable early learning and development skills?
Differences found between groups at baseline may be controlled for in follow-up study planned for April 2016
Few differences gives more confidence in our comparison of children’s learning and development growth during school year
To what extent are teachers able to identify the social & emotional assets and resilience in the preschool sample? Are there differences between the four groups?
What can be learned about children’s psychosocial development from their drawings and responses to open ended-narrative responses?

4. II. Sample 553 five-year-old children in Zomba, Malawi
All children in this study are enrolled in early childhood development centers (CBCCs) and are divided into four groups:
Save the Children’s Early Literacy and Mathematics program (ELM): 10 centers and 139 children
Save the Children’s HEART program: 10 centers and 135 children
Both ELM and Heart program: 10 centers and 140 children
Previously established early childhood program (Control): 10 centers and 139 children
14 children were randomly chosen at each CBCC
14 children were not available at some smaller centers
All children were given the opportunity to decline participation without penalty

5. II. Sample Table 1: CBCC Sample, by Group
Comparison
ELM Only
HEART Only
Heart & ELM
Year Established (Average)
2011
2013
2010
Total Enrollment 49 47 60 56
Boys
48%
49%
45%
Girls
52%
51%
55%
Sampled as evenly as possible from both genders: Boys (47%) and Girls (53%). Sample reflects total enrollment at all CBCCs – slightly more girls than boys.
ELM Only group established, on average, more recently than the other groups

6. III. Tools used to measure outcomes
IDELA (International Development and Early Learning Assessment), with added socio-emotional items
The core IDELA assessment has 22 child-reported items covering 4 areas: Motor development, early numeracy, early literacy, and socio-emotional development. 20 of these were used.
3 additional items (strengths, preferences, and free drawing) were added to learn more about children’s socio-emotional development
1 cognition item related to executive functioning was also added (memory)
SEARS-PRE (Socio-Emotional Assets and Resiliency Scale for Preschool)
25 teacher-reported items adapted by Dr. Girija Kaimal from Dr. Nancy Ravitch’s original work
Teachers rated children’s behavior in class on a 3 point scale from Rarely to Often
For example: Child smiles when playing with other children and adults

7. IV. Motor skills scores, by group
Table 2. Baseline Motor Development
Points Possible
Comparison
ELM Only
HEART Only
Heart & ELM
Drawing a Human 8
37%
34%
28%
41%
Folding 4
40%
39%
44%
Copying a Triangle
32%
38%
35%
46%
Total Motor Index 3
36% N
139
135
140

8. IV. Motor skills scores, by group
On average, children completed an average of 1.1 out of 3 motor items correctly (38% correct)
No significant differences between the groups
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

9. IV. Early numeracy skills scores, by group
Table 3. Baseline Early Numeracy
Points Possible
Comparison
ELM Only
HEART Only
Heart & ELM
Measuring Size/Length 4
78%
76%
84%
83%
Sorting 2
50%
48%
46%
Shape ID 5
26%
29%
28%
Number ID 20 2% 3% 5%
Counting 3
27%
23%
24%
Simple Operations
47%
45%
42%
54%
Puzzle Completion
16%
15%
20%
21%
Total Numeracy Index 7
35%
34%
37% N
139
135
140

10. IV. Early numeracy skills scores, by group
On average, children answered an average of 2.5 out of 7 mathematics questions correctly (36% correct)
No significant differences between the groups
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

11. IV. Early literacy skills scores, by group
Table 4. Baseline Early Literacy Skills
Points Possible
Comparison
ELM Only
HEART Only
Heart & ELM
Print Awareness 3
48%
51%
47%
57%
Letter ID 20 1% 2%
Expressive Vocab
28%
27%
25%
32%
Oral Comprehension 5
45%
41%
Phonetic Awareness 7% 6% 9%
Writing 4
29%
33%
Total Literacy Index 6
26%
30% N
139
135
140

12. IV. Early literacy skills scores, by group
On average children had an emergent literacy score of 1.6 out of 6 possible points (27% correct)
No significant differences between the treatment groups and the comparison group
The Heart & ELM Group scored significantly higher the HEART Only group
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

13. IV. Socio-emotional skills scores, by group
Table 5. Baseline Socio-emotional Development Skills
Points Possible
Comparison
ELM Only
HEART Only
Heart & ELM
Personal Information 4
49%
48%
50%
54%
Friends 10
47%
44%
Empathy 5
34%
41%
Solving Conflict 3
32%
31%
33%
Total Socio-Emotional Index
42%
43% N
139
135
140

14. IV. Socio-emotional skills scores, by group
On average children had a score of 1.7 out of 4 possible points (43% correct)
There were no significant differences between groups
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

15. IV. Overall IDELA score, by group
On average children had an IDELA score of 6.9 out of 20 possible points (36% correct)
No significant differences between groups
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

16. IV. SEARS-PRE scores, by group
On average teachers rated children with a SEARS-PRE score of 57 out of 74 possible points (77%)
There were no significant differences between any of the groups
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

17. IV. SEARS-PRE – IDELA Relationships
Overall there is a significant, but small, relationship between teachers’ ratings of children’s socio-emotional development and IDELA scores. This relationship is found for the total IDELA score, and the Motor and Numeracy sub-Indices
This result is driven by children in the ELM Only group. No significant relationships exist between the teachers’ rating and the IDELA scores for the other groups.

18. V. Conclusion and Next Steps
Conclusions
Only one significant difference between the groups (HEART & ELM scored significantly higher than HEART Only on emergent literacy sub-index), but none on total IDELA score
Baseline scores will be controlled for in follow-up analyses
Few significant differences validates the selection of the groups, especially the control, and sets up for a clear endline & gains analysis
Next Steps
Follow-up study scheduled for June 2016 to measure gains and assess impact
Close monitoring of teacher trainings, inputs, and other project activities of all sample schools (including comparison) in the meantime

19. Lauren Pisani (lpisani@savechildren.org)
Cindy Sobieski

20. Findings from SEARS-Pre and IDELA open-ended responses

21. Purpose of open-ended responses
Are helpful in new/ pilot studies to identify context-specific considerations
Can help identify unique, unidentified & relevant information about the population being studied
Can corroborate, explain, and enrich the quantitative data
Identify new questions and/ or categories of relevance to the study

22. SearsPre: Teacher Open Response on Student
Table 6. SearsPre Teacher Open Response on Student
Comparison
ELM Only
HEART Only
Heart & ELM
At Least One Comment from Teacher
65%
82%
76%
Student Challenges:
Family Adversity
29%
22%
24%
32%
Physical Disability 3% 4% 2% 7%
Learning Disability 9% 8%
Emotional Difficulties
21%
15%
19%
Often Absent 1%
Student Strengths/Support:
Family is Committed to School
37%
41%
39%
48%
Academic Strengths
51%
55%
60%
Emotional Strengths
44%
54%
59%
53% N
139
138
131
No statistically significant differences

23. Sears-Pre Open-Ended Responses: Student Challenges
Family Adversity most common, followed by Emotional Difficulties
No significant differences between the groups

24. Sears-Pre Open-Ended Responses: Student Strengths & Family Support
Across all groups, teachers had many positive things to say
No significant differences between the groups
Similarity indicates likely comparability of the groups, good for the endline analysis

25. Analytic strategy for Free Drawing
Set of responses was reviewed to identify overall themes
Each non-response was coded as a 0
Each response was coded by a number assigned to recurring themes in the responses
Frequencies were then tallied for each question by group

26. Completion of the Free Drawing
Children that completed both the drawing and were able to describe it
Heart & ELM group drew significantly more than HEART Only group
Several students in all groups declined to do the free drawing (16% in Comparison, 7% in ELM Only, 15% in HEART Only, 5% in Heart & ELM)
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001

27. Themes in the free drawings
Table 7: Free Drawing Themes, by Group
Comparison
ELM Only
HEART Only
Heart & ELM
Buildings (House, School) 5% 6% 7% 4%
People (Boy, Girl, Mother, Friend)
20%
32%
37%
29%
Nature (Trees, Animals)
22%
19%
14%
Letters/Numbers/Shapes
12%
Vehicles/Roads (Car, Bus, Airplane)
16% 9%
Food 1%
Things (Ball, Pencil, Doll)
11%
Other 0%
Did Not Draw
15% N
139
135
140
People and Nature the most common items drawn

28. Listing Preferences Table 8: Number of Preferences Listed, by Group
Comparison
ELM Only
HEART Only
Heart & ELM
# of Items Named that Child Likes to Eat
3.8
4.0
3.5*
4.1
# of Items/Games Child Likes to Play With/Play
1.6*
1.8
1.4**
2.1 N
139
135
140
Note: Asterisks relate to group with the highest score;
* p <.05, ** p < .01, *** p < .001
Generally, children named more items they liked to eat than games they liked to play/toys they liked to play with
Children in the HEART & ELM group named significantly more food preferences than children in the HEART Only group
Children in the HEART & ELM group named significantly more game/toy preferences than children in the HEART Only group and children in the Comparison group

29. Themes in Strengths Table 9: Strengths Named, by Group Comparison
ELM Only
HEART Only
Heart & ELM
Strengths Named:
Reading/Writing
10%
12%
16%
18%
Dancing
11% 8%
Sports/Athletics
37%
31%
36%
Reflective/Gets Along Well W/Friends
21%
13%
20%
Helping at Home (Cooking, Farming)
26%
24%
35%
# of Strengths Named
1.4
1.5 N
139
135
140
Only top 5 most common strengths listed; No significant differences between the groups
Overall, 9% of children could not name any strengths. There were no significant differences between groups.

30. Next Steps Endline analysis in June 2016
Will allow for comparison of teacher responses between groups, and how closely they are connected to actual student performance
Can compare gains in response rates, and in the number of responses, across groups
Can look for meaningful shifts in qualitative data between groups.