1. Quasi-experimental Evaluation Design for the NECT Learning Programme
SAMEA Conference
27 October 2017

2. Background to the NECT Learning Programme

3. Background to the NECT
In line with the NDP, the Education Collaboration Framework is a partnership between government and key stakeholders (business, labour and civil society) to increase collaboration and improve outcomes
The National Education Collaboration Trust (NECT) was established to operationalise the framework and oversee the implementation of programmes
The District Improvement Programme (DIP) is the NECT’s flagship programme and accounts for the vast majority of its expenditure
The DIP leverages more than R160m per annum and is implemented across all schools in 8 districts in the country (and growing)

4. What is the District Improvement Programme?
The Learning Programme is the “common core” of the DIP; focussed within these components
Teacher Professionalisation
Standardised programmes, content knowledge and fundamentals of performance.
Courageous and Effective Leadership
Resources management, curriculum tracking and support. Courageous leadership module as running thread
Capacity of the State to Deliver
Training and coaching to drive and maintain education change.
System level innovations
Learner Welfare and Support
Psycho-social health assessment test. Reigniting the role of Council of Learners in education and development.
Parent and Community Participation
Community members , unions, business member District Steering Committee overseeing education improvement in 8 districts. Volunteerism in schools.
Source: NECT (2016)

5. What is the Learning Programme?
The Learning Programme has three key levers:
Detailed Lesson Plans and Curriculum Trackers
High Quality Materials
Instructional Training and Coaching
The key focus of the Learning Programme is to improve:
Curriculum management and oversight
Curriculum coverage
Assessment practices

6. Evaluation methodology

7. Evaluation Methodology: Overview
Impact evaluation
Focussed on Foundation Phase teaching and learning
EGRA and EGMA type learner assessment tools in 150 schools
Grade 1 assessment in early 2017 and Grade 2 end of 2018
Supplement by use of learner workbook data
Supported by a process evaluation
Semi-structured interviews at national district and school level
In-school data collected by evaluation team
NECT monitoring data
Case studies in a sub-set of treatment schools

8. Choosing a quasi-experimental approach
Key to impact evaluation is establishing a credible counterfactual
The big challenge is to remove (or at least minimize) unobserved biases
An RCT attempts to remove bias by randomising treatment in advance
However, the Learning Programme started in 2015, while the evaluation only started in 2017
Randomisation therefore not possible
A quasi-experimental approach was therefore selected: matching

9. What is matching?
Matching involves constructing a comparison group that is sufficiently similar to the treatment group at the outset of the programme based on observable characteristics
In contrast to an RCT, this comparison group is typically selected ex-post
Since the programme is mainly administered at school level, we select comparison schools that are sufficiently similar to a set of sampled treatment schools that received the programme; using DBE data available on these schools
School level because the programme is implemented at this level

10. Approach to matching
Initially considered choosing comparison schools from within treatment districts:
However programme was expanded across all schools in treatment districts (universalisation)
As it is a district level programme, there would also have been within district spill-overs through e.g. district office intervention
This meant that schools had to be selected from other districts
Matching would thus involve:
First selecting comparison districts
Then selecting comparison schools from within comparison district
School level because the programme is implemented at this level

11. Selecting comparison districts

12. Focussed District Model
District selection
Focussed District Model
Whole District Model
Vhembe
The Learning Programme comprises two different implementation models:
The Whole District Model (WDM)
The Focussed District Model (FDM)
Programme is implemented by 3 different lead agents
One WDM district and two FDM districts used as treatment districts based on a number of criteria (next slide)
First step is looking at potential comparison district for each treatment district
Waterberg
Bohlabela
Bojanala B
uThungulu U P
Pinetown M
Mt Frere L
Libode
The Learning Programme c
Source: DBE (2013) Education district in South Africa: A review

13. District selection - factors
Tshipisi Sagole
Mogala-kwena
Mopani
Vhembe
Waterberg
Availability of appropriate comparison districts:
School profile and performance
Language profile of district
Socio-economic similarities
Limited spill-overs and other programmes
One district from each lead agent
Composition of treatment district:
Relatively homogenous language profile
Quintile distribution of schools in the district
Lebowa-kgomo
Bohlabela
Sekhukhune
Ngaka
Modiri Molema
Bojanala
Ehlanzeni
Dr KK
Umkhanyakude U
Zululand
Uthungulu
Umgungundlovu
Ilembe P
Pinetown U
Umlazi
Mt Frere M L
Mbizana
Libode
Lusikisiki
Source: DBE (2013) Education district in South Africa: A review

14. Example 1: Bohlabela (language)
Mopani
No good language matches in Mpumalanga
Mopani is in Limpopo – cross province comparison not ideal
Hence no good matches
Bohlabela
Ehlanzeni
Umkhanyakude
Source: DBE (2013) Education district in South Africa: A review

15. Example 2:Vhembe (spill-overs)
Tshipisi Sagole
Mogala-kwena
Vhembe
Waterberg
Tshipise Sagole (TS) relatively similar to Vhembe across language and quintiles
However project managers reported significant spill-overs between districts
Used to be a single district
District management still integrated
Project materials have been used in TS
No other district has sufficient Tshivenda schools to act as counterfactual for Vhembe
Lebowa-kgomo
Sekhukhune
Source: DBE (2013) Education district in South Africa: A review

16. Example 3: Eastern Cape
Districts broadly similar across key characteristics
Mt Frere seen as having stronger implementation than Libode
Mbizana selected as comparison district for Mt Frere
Mt Frere M L
Mbizana
Libode
Lusikisiki
Source: DBE (2013) Education district in South Africa: A review

17. Districts selected Treatment district Comparison district Language
Bojanala
Dr Kenneth Kaunda
Setswana
Mount Frere
Mbizana
isiXhosa
Uthungulu
Ilembe
isiZulu
Bojanala
Dr KK P
Pinetown U
Umlazi
Mt Frere L
Lusikisiki
Source: DBE (2013) Education district in South Africa: A review

18. Selecting comparison schools

19. Matching: School level (1)
The matching process was to:
Select a random sample of treatment schools
Select matching comparison schools based on this sample
Ideally, comparison schools should be selected using the criteria used to select schools into the programme.
However programme selected was not based on objective criteria that could be replicated using available data

20. Matching: School level (2)
The data fields used were selected based on three main factors:
Data must be available across all schools
Fields (such as quintile) that were seen as good proxies for the original programme selection criteria
Fields that are indicative of school performance at the baseline of the programme (2014)

21. Matching methodology (1)
A variety of different matching approach exist, e.g.:
Propensity Score Matching (PSM)
Exact Matching
Matching based on distance metrics
PSM regressions yielded few statistically significant parameter estimates and a very narrow band of propensity scores
Hence a hybrid approach between exact matching and Euclidean distance matching was used

22. Matching methodology (2)
In particular, exact matching was performed to stratify schools into groups across the following fields:
School quintile (1, 2 or 3)
School type (Primary or Combined)
Grade 1 enrolments, grouped into 4 categories
Within the above categories, nearest neighbour matching was then performed by minimising the Euclidean distance measure of each school’s average 2014 Annual National Assessments marks across Grade 1 and 3 mathematics and language

23. Thank you!