1. 1 Improving Education Management in Madagascar: Results and Implications of an Impact Evaluation Gérard Lassibille, Jee-Peng Tan, Cornelia Jesse and Team AGEMAD Africa Program for Education Impact Evaluation (APEIE) Second Workshop, Dakar, Senegal Dec 17-19, 2008

2. 2 Outlline Country context Current management processes AGEMAD tools and procedures Design of the experiment Implementing the impact evaluation Key results and implications Scaling up and institutionalization

3. 3 Country Context Education viewed as central to government reforms to accelerate growth and reduce poverty Key recent measures in primary education include: Restructuring of the primary and secondary cycles of schooling Introduction of new pedagogical approaches Reduction of grade repetition Distribution of school kits to all primary school children Increased hiring of contractual teachers Use of school grants to improve service delivery and performance AGEMAD set in the context of these reforms: To improve understanding of management issues in education To support the ongoing and future reforms

4. 4 Evidence of progress in primary education: Increase in primary education’s share of total spending on education: 59% in 2006, up from 38% in 2002; 4 million in school in 2006, up from 1.7 million in 1996 59,000 teachers in 2006-7, about twice the number 10 years ago

5. 5 Yet, many problems and challenges remain … Evidence of weak performance: ½ of each cohort of 1 st graders don’t finish the primary cycle; Repetition rate still high at 18% in 2005 (30% in 2000) Low test scores: in 2004-5 PASEC, average score of 50% in Maths and Malagasy and 32% in French; deteriorated since 1997-98 More than ½ of teachers are contractual teachers; poorly trained and poorly paid Multiple causes of weak sector performance: Some beyond control of the sector Some depend on decisions within the sector, including acting to reduce: Inconsistencies in teacher allocation across schools; Poor utillization of instructional time; Ineffective management of the pedagogical processes at the school and classroom levels.

6. 6 Current Administrative Structure Actors in the System, 2006-7 PupilsTeachers School DirectorsChefs ZAPChefs CISCODRENMinistry Total number 3,103,00059,00017,6001,544111221 Number under each actor: Pupils 531772,01027,900141,0003,103,000 Teachers 338530269059,000 School directors 1116080017,600 Sub-district officer (ZAP) 14701,544 District officer (CISCO) 5111 Regional officers (DREN) 22

7. 7 Evidence of Weak Management …. Many aspects of the pedagogical processes are poorly managed and tasks essential for student learning are neglected Poor monitoring of pupil absenteeism: Only 13 days a month, on average, monitored by teachers Absences not supervised by school directors: 10% never monitor them 1/3 of attendance records not signed off by directors Not a topic of discussion with teachers in 80% of cases

8. 8 Evidence of weak management (contd.)… Neglect of basic pedagogical tasks: Among teachers: 20% don’t prepare daily lesson plans Only 15% consistently prepare daily and bi-weekly lesson plans Among school heads: 1/3 never discuss with teachers their daily lesson plans ½ fail to sign off on their teachers’ lesson plans 70% never sign off on teachers’ daily lesson plans

9. 9 Evidence of weak management (contd)….. Poor communication on student learning: Results of tests and quizes are poorly recorded, if at all; 25% of teachers don’t prepare individual student report cards; Communication from teachers to parents is often perfunctory; Pupil absenses is poorly communicated to parents. School directors hardly involved in following-up on student performance: 3/4 don’t discuss issues with learning outcomes with their teachers; only 20% sign-off on test results and student report cards)

10. 10 Evidence of weak management (contd.)….. Inadequate supervision of teacher absences : Absenteeism rate of nearly 10% Only 8 % of school directors follow teacher absences closely taking daily attendance; monitoring and posting a monthly summary of absences More than 80% of directors fail to report teacher absences to administrators at the sub-district and district levels

11. 11 AGEMAD tools and procedures Objective Strengthen management of service delivery in primary education How? Modifying behavior of key actors by inserting supervision and follow-up at key points in the adminstrative hierarchy Making explicit to the actors their responsiblities and supporting them with the tools to accomplish their tasks

12. 12 AGEMAD tools and procedures More than 30 tools developed for tasks considered essential for a well-functioning system: 7 for teachers 8 for school directors 8 for sub-district administrators 9 for district administrators Tools targeted following areas: Pedagogy Student learning and follow-up Management of instructional time Administration School statistics Partnership with the local community

13. 13 AGEMAD tools and procedures AGEMAD tools and procedures: Most tools are not new Should be used by each actor at specific times during the year School report cards for school directors, sub- district and district levels officers: Complement the AGEMAD tools and procedures Draw attention to schooling outcomes Include comparative data, allowing a school to compare its outcomes with those of other schools Serve as basis for dialogue and accountability

14. 14 Setting up the experiment Objective: Evaluate impact of the AGEMAD interventions on service delivery and schooling outcomes Experiment seeks to answer three key questions: Do AGEMAD interventions work? Which particular AGEMAD package worked? Were there spillover effects? Experiment will run for two school years, 2005-6 to 2006-7 Support from MIT Poverty Lab for experimental design

15. 15

16. 16

17. 17 Implementing the experiment Report cards of schools, sub-districts and districts: Developed by a team in the EMIS section of the Ministry of Education. A total of 10,000 report cards were produced. Tools, operational guides and training: 200,000 tools et 11,000 guides produced and distributed Training provided to nearly 4,000 participants 1 st year: 4-day course for the sub-distrcit and district officiers; 2-day course for teachers and school directors; 2 nd year: 2-day course for the sub-district and district officiers; 1-day course for the school directors and their teachers; Training for teachers and school directors provided by sub-district officers School meetings: Over the two years, 1,500 meetings tool place with support for organization and facilitation provided by specially recruited, to discuss elaboration of school improvement programs

18. 18 Implementing the experiment Implementation team: Stable team of about 15 MOE staff AIDE et ACTION AFD (2 technical assistants on site) World Bank Support from more than 50 persons Financing from the WB, AFD, MOE, Ireland, and the EFA-FTI EPDF

19. 19 Collecting the data School surveys: Impromptu school surveys (1,200 schools and 4,000 teachers) Survey of the sub-district and district administrators Artifacts of service providers’ work: Administrative and pedagogical tools used by teachers and school heads (at the end of the experiment, in 40 schools and 850 artifacts). Achievement tests (incl. pupil surveys): Year 1: tested about 25,000 pupils in Grade 3; Tear 2: tested about 22,000 pupils in same cohort (i.e., Grade 4) Maximum of 25 pupils per school Administrative data: Pass rate on the end-of-cycle examination.

20. 20 Analytical approach Two types of AGEMAD impact expected: Direct impact on the behavior of the actors Indirect impact on student achievement Evaluating the indirect impact is easy: simply compare across the treatment and control groups Evaluating the indirect impacts presents more difficulty: The simplest (or simplistic): compare task by task across across the treatment and control groups Complication 1: multiple actors; focused on school personnel (director and teachers) Complication 2: multiple tasks per actor; focused on tasks considered essential by Malagasy educators

21. 21 Analytical approach Teacher: Takes daily roll call Prepares daily lesson plan Makes two-monthly plans of lessons Monitors student learning Tested pupils during the past two months Helps lagging students Discusses student learning issues with the director School director: Keeps a register of enrollments Signs off on daily roll call Analyzes student absences on a monthly or bi-monthly basis Reviews pupils’ test results Takes stock of teacher absences Informs sub-district or district officer about teacher absences Follows up with teachers on lesson planning Seven essential tasks:

22. 22 AGEMAD interventions seeks to influence key actors to discharge their key responsibilities: Definitions adopted for the analysis:  A good teacher performs all 7 essential tasks in TORs  A good director executes all 7 essential tasks in TORs  In a well managed school all the staff execute all their essential responsibilities Analytical approach

23. 23 Results—Impact on Provider Behavior Control schools Diffusion of AGEMAD tools and procedures Direct to schools Via sub-district & district officers Via district officers Teacher absenteeism rate (%) 9.28.79.910.1 % of teachers performing all 7 tasks 42.463.0*53.653.4 % of schools where all the teachers perform all 7 essential tasks 22.142.9**33.833.2 % of schools where the director and all the teachers perform their essential tasks 15.236.5**23.224.6 ** and * signify differences are statisticaly significant at, respectively, 1% and 5%.

24. 24 Results: Impact on schooling outcomes Control school Diffusion of AGEMAD tools and procedures Direct to schools Via sub-district & district officers Via district officers Attendance rate (%)86.690.7*88.189.6 Repetition rate (%)22.617.5*20.018.1 Dropout rate (%)6.15.5 4.3 Pass rate at CEPE (%)61.973.076.376.7 Year Two test scores Math49.451.249.248.6 Malagasy48.550.949.548.9 French29.430.029.229.1 All three subjects41.943.542.141.7 *=difference is statistically significant at 5%.

25. 25 Implications of the results Who to target? Prioritize school-level actors “Cascade” model alone, as currently defined, doesn’t work What’s in the intervention package? Materials in the form of tools and guides; Training actors in using the tools and guides; On-site support from facilitators; and Community-school meetings around school report cards, as basis for school improvement plans What are the costs? Cost of intervention per school high during impact evaluation Costs during impact evaulation NOT necessarily cost to scale up

26. 26 Implications for scaling up and instutionalization Mainstream IE results into MoE activities Need a champion from the start Need early involvement of a national team, with good technical support Necessary to sustain change in actors’ attitudes & behaviors Use existing structures and mechanisms for scale up: Tools, guides and training modules integrated into teacher training Tool distribution, training and facilitated school meetings funded through the local catalytic funds based on regional, district and school performance plans and needs Develop leaders to drive change in management practices Discussion underway on collaboration in leadership training between Madagascar MoE and partner organization in another country

27. 27