1 Impact Evaluations: Experiences from Sierra Leone and Ghana

2 Country Summary Sierra LeoneGhana PoliticalPost ConflictRelative stability GovernanceMII: 37 th (48 SSA)MII: 7 th (48 SSA in 2007) EconomicGDP/C: US$286 PPP: US$806 GDP GR: 4.5% (2005) GDP/C: US$650 (2007) PPP: US$2,480 (2005) GDP GR: 6% (2006) Poverty70%28% Human Dev ’ tHDI rank: 177 th (177)135 (177 in 2005) Population6 million23 million (2008) Education5,000 pri. Schools 1.3 m enrollment 15,200 pri. Schools 3.1 m enrollment (2007) IE QuestionTextbooks and TTSMCs

3 Key Evaluation Questions SSierra Leone DDoes textbook distribution improve basic pupils ’ learning outcomes? DDoes the training of teachers in the use of textbooks improve pupils ’ learning outcomes? GGhana WWhat is the Impact of School Management Committees (SMCs) on learning outcomes at basic level?

4 Background to Key Evaluation Questions SSierra Leone SSignificant expenditures on textbooks --Target of 1:1 is yet to be reached even in intervention areas after several years of projects TTeacher in-service training is very important --An estimated 40% of Sierra Leone ’ s 35,000 to 40,000 teachers are untrained and unqualified

5 GGhana SSignificant improvements in Primary Completion Rate has led to questions of how to improve quality of basic education CCommunity and school based interventions have become very critical with the roll out of decentralization in the management and governance of basic schools.

6 Evaluation Design Sierra Leone  Four Local Councils out of 19 LCs were selected each representing one of the four regions in Sierra Leone — Local Councillors were invited to a meeting to randomly select by lottery LCs who will participate in the Impact Evaluation.  Schools were randomly selected into three groups from the EMIS data as follows: -- control (30 primary schools), -- textbook treatment (30 primary primary schools), -- textbooks and teacher training treatment (30 primary schools)

7 Evaluation Design Sierra Leone  Pupils in Primary 4 and 5 were selected for the IE because they could read. Primary 6 was left out because they are preparing for the NPSE and it was agreed that the IE should not be intrusive. --Baseline concentrated on achievement test scores for P3 and P4 because these cohorts would move to P4 and P5 by the time of the actual evaluation.

8 Evaluation Design Ghana  53 deprived districts are already receiving additional resources from government— these districts were selected based on a number of indicators spanning inputs, access, quality of learning.  212 public primary schools were selected randomly from these 53 distircts.

9 Evaluation Design Ghana  Three groups were randomly obtained from these 212 schoolsbased on the EMIS as follows: --control group (70 basic schools) --information about SMC roles and responsibilities group (71 basic schools) --information + capacity building of SMCs group (71 basic schools)

10 Components of a Successful IE  Start-up workshop was/is critical—Abuja, Dakar  Field Coordinator—whose key focus on IE  Government ownership and commitment, Ministry of Education involvement at concept stage—but remember to diversify involvement  WB involvement; TTL’s role in trying to coordinate all stakeholders

11 Elements of a Successful IE  Timely availability of funds (EPDF is great!)  Follow-up vigorously on the intervention budget  Understand the issues and take risks  Keep impact evaluation on the radar at all times—take every opportunity to communicate effectively on IE

12 Issues and Risks Watch out for possible derailment of the process and be prepared for this e.g. changes in the socio- political environment; problems with funding the intervention…  Political changes, changes in government  Ethical concerns, who gets textbooks first?  Manage potential bad press  Manage the expectations, because we may find “nothing”  Create the stage for the next IE

13 Political Change-Sierra Leone  Change in government in Sierra Leone --New councilors were elected last August, thank goodness they thought to involve the more permanent Chief Administrators in the district and school selection workshop  Commission of Inquiry in Sierra Leone --All activities have sort of come to a stand-still because of appearances before the commission. Textbooks distribution in Pujehun by the MCSL and RADA have halted because of appearances before the commission.

14 Political Change-Ghana  Elections in Ghana --All sector resources are understandably concentrated on the elections --First round was on Dec. 7, run-off on Dec. 28.  Critical implications for interventions budget --A new government means a new budget, new priorities etc.

15 Status of Impact Evaluation  Sierra Leone --EPDF application and funds obtained --Field Coordinator is on board --Recruited Statistics Sierra Leone for data collection, data entry --Textbooks are still being distributed to treatment schools --Teacher Training Manual is being revised

16 Status of Impact Evaluation  Sierra Leone—next 12 months --Complete textbooks distribution --Complete manual revision --Roll out teacher training; Roll out textbooks --Plans to have another small survey, probably testing teachers as well --Obtain the intervention budget—EFA FTI; GoSL --Data collection, analysis for follow-up survey --IE Report

17 Status of Impact Evaluation  Ghana --EPDF application and funds obtained --Field Coordinator is on board --Questionnaires have been finalized --Recruitment of firm for baseline is far advanced—technical and financial evaluation reports being finalized

18 Status of Impact Evaluation  Ghana—next 18 months --Baseline survey planned for Jan Data entry and analysis Jun Training of trainers for SMCs --Training of SMCs, capacity building

19 Status of Impact Evaluation  Ghana—next 18 months --Mid-Term Review; classroom observations and surprise visits to measure pupil and teacher attendance --SMC survey; --Final Data Collection, data entry and analysis --IE Report