1. The Midline Evaluation September 2016

3. Motivation behind this project
Reading is at the heart of the quality challenge in SA
Reading is key to this – gateway to all other learning
58% of children not learning to read by grade 4.
Root cause of school dropout in grades 10-12
Early interventions are most cost-effective
Strategic and symbolic focus on reading
Really a literacy intervention
Robust impact evaluation using RCT

4. Our research agenda
What interventions (built on evidence from elsewhere) make an impact in the SA context?
What is the cost-effectiveness of each intervention?
What differences exist in the “pre-reading abilities” of children at the start of school?
Girls and boys
Home characteristics
Urban and rural schools
How do reading inequalities evolve over time?
What levels of reading fluency in grades 1 and 2 are required for successful learning in later grades?

5. Project design
Goal: To improve the acquisition of reading in home language (Setswana).
230 schools in the North West province:
Districts of Dr Kenneth Kaunda and Ngaka Modiri Molema
Quintile 1-3 schools using Setswana as LOLT
Project duration is 2 years: following Grade 1 cohort (2015) into Grade 2 (2016)
We administer 3 interventions and evaluate their impact relative to a valid control group
Baseline survey Jan/Feb 2015;
midline Nov 2015;
endline Nov 2016
Survey includes testing learners, parent questionnaire, teacher questionnaire, principal questionnaire

7. Description of Interventions
230 schools in North West

8. Instructional Core

9. Description of Interventions
230 schools in North West
Per learner costs
Triple cocktail: Expensive for national roll-out, but R7m – R8m for one grade in 100 priority schools.

11. Descriptive analysis of baseline and midline (mostly) results

12. Interesting to note from baseline
An aging teacher population
1 in 4 teachers were 56 or older
10% younger than 40
Girls substantially outperform boys at the start of school
Parent characteristics predict school readiness (including taking responsibility for child’s education)

13. Baseline balance tests

14. Midline test instruments
EGRA Letter Sound Recognition
EGRA Word Recognition
EGRA Non-Word Decoding
Sentence Reading
Paragraph Reading
Writing/dictation
Phonological Awareness.

15. Summary statistics for each sub-test in Midline learner assessment
min
max
p10
p25
p50
p75
p90
Letter recog.
110 4 16 38 54
Word recog. 50 3 9 22
Non-word decoding 6 18
Sentence reading 11 1
Paragraph reading 64 30
Comprehension
Writing 12 8
Phonological awareness
Combined score
-0.943
3.650
-0.868
-0.718
-0.444
0.486
1.693

16. Attrition
Reason given for attrition

17. What does predict attrition?

18. Main Results by end of Grade 1 (3 terms of intervention)

19. Letter recognition gains

20. Letter recognition gains

21. Word recognition (Midline)

22. Non-word recognition (Midline)

23. Sentence reading (Midline)

24. Paragraph reading (Midline)

25. Comprehension (Midline)

26. Writing/Dictation (Midline)

27. Phonological awareness (Midline)

29. Model specification Predict outcome (e.g. Combined score)
Key explanatory variable is Treatment group
Controls for:
Baseline learner scores
Learner gender
Parent education
School ANA performance in 2014
Community poverty (from Census 2011)
Rationale: Control for incidental prior differences; improve precision

30. Main regression results

31. Effects on sub-tests

32. Regression results (impact on writing)

33. Heterogeneous treatment effects aka Sub-group effects (Very preliminary)

34. Learner-level heterogeneous effects
Treatment 3 appears to have had an impact amongst the strongest learners at baseline
More involved parents attended?
The positive impact for Treatments 1 and 2 is clear for boys (about 0.18 SD); less clear for girls
Helps boys catch up to girls?

35. School-level heterogeneous effects
Large impact in urban schools of T1 (0.32 SD) and T2 (0.38 SD); possibly zero impact in rural schools
Large impact of T1 and T2 in Quintile 2 & 3 schools; possibly zero impact in Quintile 1 schools
Clear impact of T1 (0.17**) and T2 (0.16**) in monograde schools; possibly zero impact in multi-grade schools

36. Teacher-level heterogeneous effects
Teacher absenteeism: Impact of all 3 treatments higher amongst teachers less commonly absent
T1 (0.21**); T2 (0.18*); T3 (0.13)
Teachers who at baseline said they group children according to reading proficiency (“streaming”) benefited more from T1 and T2

37. Impacts on intermediate outcomes
Teachers in T1 and T2 were more likely to “stream”, compared to the control group.
T2 teachers conducted individualized reading assessments of learners more frequently than the control group.
Some evidence of increased reading resources in treatment 1 and 2 classrooms, especially of Setswana posters.
More exercises of all types (including drawing pictures), of written exercises, and of full sentence writing exercises in both T1 and T2.

38. Conclusions
Treatment 3 had low attendance, and hard to measure effect for those whose parents attended
Clearest effects observed for Treatment 2, and to some degree Treatment 1
Magnitude of the effect is debatable
More of an effect for boys, urban schools, higher quintile schools
Evidence of changed classroom practice and learner work – to be explored in qualitative study
No big policy warrant yet

39. Forthcoming Endline testing Oct/Nov 2016
Qualitative survey (60 schools) – October
Case studies
Extension of Treatment 1 and/or 2 in grade 3 in 2017

40. EGRS 2

41. Additional slides

42. 458 registered primary schools with enrolments in grades 1-4
Apply a series of exclusions
Exclude schools not using Setswana as language of instruction
Exclude small schools and large schools
Exclude schools with missing ANA data
affluent schools (quintiles 4 and 5)
exclude 8 pilot schools
exclude replacement schools
exclude problem schools identified by PED
Sampling Frame of 230 schools
Create 10 strata by school size, school socio-economic status and ANA performance
Randomly assign schools within each stratum to T1, T2, T3 and Control
5T1 5T2 5T3 8C
This yields 4 treatment groups
T1: Teacher training (50 schools)
T2: Coaching (50 schools)
T3: Parent involvement (50 schools)
Control group (80 schools)