STRENGTHENING ACCOUNTABILITY IN PRIMARY EDUCATION IN MADAGASCAR: RESULTS OF AN IMPACT EVALUATION Jee-Peng Tan & Cornelia Jesse, HDNED Chief Economist Office & Results for Development Institute Seminar “Demanding Good Governance - Inside & Out” March 25, 2010

Primary Education in Madagascar: Much Progress, but still Many Challenges  Signs of progress:  Primary completion rate doubled from 35% in1999 to 71% in 2008  Evidence of weak sector performance:  ½ of each cohort of 1 st graders does not finish the primary cycle;  Repetition rate still high at 18% in 2005 (30% in 2000)  Low quality: in PASEC, average test score of 50% in Maths and Malagasy and 32% in French; deteriorated since  Multiple systemic causes :  Inconsistencies in teacher allocation across schools;  Ineffective management of pedagogical processes at school and classroom levels

A Problem of Internal Accountability Tasks essential for student learning are neglected:  Inadequate supervision of pupil and teacher absenteeism e.g. more than 80% of directors fail to report teacher absences to administrators at the sub-district and district levels  Neglect of basic pedagogical tasks e.g. 20% of teachers don’t prepare daily lesson plans  Poor monitoring of and communication on student learning e.g. results of student tests and quizes are poorly recorded and communicated to parents, if at all; school directors hardly follow-up on student performance: 3/4 don’t discuss learning outcomes with their teachers

What tasks are deemed essential?  Teacher:  Takes daily roll call  Prepares daily lesson plan  Prepared bi-monthly lesson plans  Monitors student learning  Has tested pupils during the past two months  Helps lagging students  Discusses student learning issues with the director  School director:  Keeps a register of enrollments  Signs off on daily roll call  Analyzes student absences on a monthly or bi-monthly basis  Reviews pupils’ test results  Takes stock of teacher absences  Informs sub-district or district officer about teacher absences  Follows up with teachers on lesson planning

Tighter Management to Improve Accountabilty Conceptual Intervention Framework & IE Design:  Workflow tools to clarify tasks and internal accountabilities;  Facilitation of meetings between school and community;  Better information flows within school and between school and community;  Structured training for teachers and school heads Leading to:  improvement in actors’ behavior through bottom up and top down accountability  better managed school  increased school quality  higher student learning

Key Questions for Policymakers  What is the impact of tighter management of processes on school functioning and student performance?  At what administrative level are management interventions the most effective (school, district or inspection level)?

Impact Evaluation Design (1) Method: Randomized experimental design over 2 school years Interventions:  Specify actors’ responsibilities & their mutual accountability the processes through:  Management Tools and Guides for key tasks (e.g. pedagogical, administrative)  Training  Focus attention on results to clarify goals through:  Report cards: School, district and inspection report cards  School meetings: Facilitated school meetings & development of school improvement plans based on school report cards

School & District Report Cards for Better Information Flow  Report cards for school directors, sub-district and district levels officers:  Complement the tools and proceses  Draw attention to schooling outcomes  Include comparative data, allowing a school to compare its outcomes with those of other schools  Serve as basis for dialogue and accountability

Impact Evaluation Design (2) Schools AGEMAD TREATMENT Schools ZAP AGEMAD TREATMENT 2 15 CISCO AGEMAD 89 ZAP AGEMAD 80 ZAP CONTROL 303 Schools CONTROL 15 CISCO CONTROL 84 ZAP NON-AGEMAD 303 Schools CISCO AGEMAD TREATMENT 1

Collecting Data Actors’ Behavior (direct effects):  Questionnaire from impromptu school visits in 1,200 schools, with information for 4,000 teachers  Questionnaires for District and Community admin. level  Collection and analysis of tools used in 40 schools (850 tools) Schooling outcomes (indirect effects):  Test scores from standardized tests in 3 subjects  National year-end school census data: flow rates, repetition, CEPE pass rate Timeline:  2 school years,  Baseline survey/test and post-intervention survey/test

Results: Effects on Actors’ Behavior AGEMAD schools Control schools Teacher absence (%) Teacher completes all key tasks (%)63.0*42.4 All teachers in school perform their key tasks (%) 42.9**22.1 Well managed schools (%)36.5**15.2 *significant at 5% level; **significant at 1% level

12 Results: Effects on Schooling Outcomes AGEMAD schools Control schools Student attendance (%)90.7*86.6 Repetition (%)17.5*22.6 Drop out (%) Success rate at CEPE exam Student test results (post-test) Math Malagasy French All subjects *significant at 5% level

Policy Implications  Prioritize school-level actors  “Cascade” training model alone, as currently defined, doesn’t work  Though results are encouraging, better management essentially entails changing peoples’ behaviors, which takes time and effort  Mainstream IE results into MoE activities  Need a champion from the start  Need early involvement of a national team, with good technical support  Necessary to sustain change in actors’ attitudes & behaviors  Use existing structures and mechanisms for scale up:  Tools, guides and training modules integrated into teacher training  Tool distribution, training and facilitated school meetings funded through the local catalytic funds based on regional, district and school performance plans and needs  Develop leaders to drive change in management practices  Discussion underway on collaboration in leadership training between Madagascar MoE and partner organization in another country

Stay tuned…Publications forthcoming  Africa Human Development Working Paper Series «Améliorer la gestion de l'enseignement primaire à Madagascar - Résultats d'une expérimentation randomisée »  Journal Article undergoing peer review «Managing for results in primary education in Madagascar: Evaluating the impact of selected workflow interventions »

It takes a village…  Government commitment:  Stable counterpart team (15 staff from MoE with coordinator)  Partner commitment:  Financial and technical assistance from AFD (via two staff)  WB team lead by Jee-Peng Tan and Cornelia Jesse, consisting of Gérard Lassibille and Trang van Nguyen (with in-country field coordinators)  Local NGO Aide et Action to assist with training  Financing: WB, AFD, MoE, EFA-FTI (EPDF), Irish Aid, Ministry of Foreign Affairs of Norway  Timeline: 2004 – 2007  Total number of people involved: 50

The Perils of Data Collection…