1. Dr Ian Goldman, Head: Evaluation and Research PSPPD PSC 26 August 2014 The Presidency Department of Performance Monitoring and Evaluation National Evaluation System

2. The Presidency: Department of Performance Monitoring and Evaluation Overall situation with evaluations  39 evaluations completed, or underway, of which 37 are NEP evaluations. 2 Improvement plans being implemented Served at Cabinet Approved reports Research underway TORs approved Preparatio n stage Stuck 821123022  Cost benefit - R50 billion of programmes evaluated (over MTEF). 10% improvement = R5 billion, annual cost of evaluation system R50 million, so very high return on investment.

3. The Presidency: Department of Performance Monitoring and Evaluation Nos of departments involved  15/46 national departments now involved in evaluations  Nos of national + provincial departments involved in evaluations risen from 13% to 23% 3

4. The Presidency: Department of Performance Monitoring and Evaluation NEP 2012/13 DeptTitle of evaluationStatus as 5 May dtiImplementation/ design evaluation of the Business Process Services Programme Final report approved. Report on way to Cabinet DBEImpact Evaluation of Grade RFinal report approved. Report been to Cabinet HealthImplementation Evaluation of Nutrition Programmes addressing Children Under 5 Final report approved and been to Cab Committee DRDLRImplementation Evaluation of the Land Reform Recapitalisation and Development Programme Final report approved and been to Cab Committee DRDLRImplementation Evaluation of CRDPFinal report approved and been to Cab Committee DHSImplementation Evaluation of the Integrated Residential Dev. Programme Evaluation underway. DHS procurement has delayed DHSImplementation Evaluation of the Urban Settlements Development Grant (USDG) Evaluation underway. DHS procurement has delayed DBEImpact Evaluation of the National School Nutrition Programme (NSNP) DBE requested to drop. Reallocated to 2014/15. 4 CompletingUnderway About to start Delays!

5. The Presidency: Department of Performance Monitoring and Evaluation NEP 2013/14 DeptTitle of evaluation PresidencyImplementation Evaluation of Government’s Coordination Systems dtiEvaluation of Export Marketing Investment Assistance Incentive programme (EMIAI) dtiEvaluation of Support Programme for Industrial Innovation (SPII) dtiImpact Evaluation of Technology and Human Resources for Industry Programme (THRIP) Military Veterans Evaluation of Military Veterans Economic Empowerment and Skills Transferability and Recognition Programme. DSTEvaluation of National Advanced Manufacturing Technology Strategy SARSImpact Evaluation on Tax Compliance Cost of small businesses COGTAImpact evaluation of the Community Works Programme (CWP) DRDLREvaluation of the Land Restitution Programme DAFFImpact Evaluation of CASP DAFFImplementation Evaluation of MAFISA DHSBaseline for informal settlements targeted for upgrading DHSEvaluating interventions by DHS to facilitate access to the city. DHSDiagnostic of whether the provision of state-subsidised housing has addressed asset poverty for households and local municipalities DPMEImpact Evaluation of the Outcomes Approach 5 Completing by Sept Underway About to start Delays!

6. The Presidency: Department of Performance Monitoring and Evaluation 2014/15 6 DeptEvaluation DEAEvaluation of the Effectiveness of Environmental Governance in the Mining Sector (EEGM) DHETDesign Evaluation of the Policy on Community Education and Training Colleges (PCETC) DHSImpact Evaluation of the Social Housing Programme (SHP) DSTEvaluation of the Indigenous Knowledge Systems Policy (IKSP) DSDDiagnostic Evaluation/Programme Audit for Violence Against Women and Children (AVAWC) DSDDiagnostic Review of Coordination of the Social Sector Expanded Public Works Programme SAPSEconomic Evaluation of the incremental investment into the SAPS Forensic Services DAFF/DRDLRImpact Evaluation of the Ilima Letsema Programme and Irrigation Schemes DAFFImpact evaluation of MAFISA (quantitative) – through 3ie DAFF/DRDLRPolicy Evaluation of Small Farmer Support DBEEvaluation of the Funza-Lushaka Bursary Scheme DBEImpact evaluation of National School Nutrition Programme DRDLRImpact evaluation of Land Restitution Programme – through 3ie DPMEImpact/implementation evaluation of the MPAT system DPMEImplementation evaluation of the dept strategic planning and APP system UnderwayTORs developed No TORs yet Delays!

7. The Presidency: Department of Performance Monitoring and Evaluation Evaluations selected for 2015/16  Impact/implementation evaluation of the evaluation system (DPME).  Implementation evaluation of citizen-based monitoring programme (DPME).  Agricultural Extension Recovery Plan (DAFF).  Evaluation of the New School Curriculum (DBE).  Evaluation of the Asset Forfeiture Unit Sub-programme (NPA).  Diagnostic evaluation of the Non-Profit Organisations Regulatory Framework and Legislation (DSD).  Implementation Evaluation of the National Drug Master Plan in addressing all forms of Substance abuse (DSD).  Evaluation of the effectiveness of the South African Post-School Education and Training (PSET) quality assurance regime (DHET).  Evaluation of Early Grade Reading in SA (DBE).  Evaluation of the Mining Charter  Evaluation of the service delivery improvement planning system 7

8. The Presidency: Department of Performance Monitoring and Evaluation8

9. 9 Programme being evaluated Outcome needed from evaluation results CloseMajor changes needed Minor changes needed No changes needed Early Childhood Development (ECD) X Business Process Services Programme X Grade R (reception year of schooling) X Nutrition Programmes addressing under 5s X Land Recapit. and Dev Prog (RECAP) X Comprehensive Rural Dev Prog (CRDP) X Export Marketing Investment Assistance X Support Prog for Industrial Innovation (SPII) X Land Restitution Programme X Government Coordination System (clusters/MinMECs/Implementation Forums) X Micro Agric Financial Institution (MAFISA) X Presidential Intervention in KSDX Many programmes need major changes

10. The Presidency: Department of Performance Monitoring and Evaluation Progress with National Evaluation System  3 National and Provincial Evaluation plans to date, last week selected evaluations for 4th  39 evaluations completed, underway (11 completed, 2 public)  18 guidelines, evaluation standards, competences, trained >600 government staff  Building demand – emphasising learning, reports to Cabinet, working with Parliament, publicising reports, training for DGs/DDGs in use of evidence  Evaluation Repository – audit of evaluations since 2006, plus new - 103 evaluations on the website  2 provincial evaluation plans (WC and GP), working with 4 more provinces, and 5 departments have departmental evaluation plans  First Annual Report on the National Evaluation System (for 2013/14) now available 10

11. The Presidency: Department of Performance Monitoring and Evaluation11

12. 12

13. 13

14. Early indications of policy impact  Role of improvement plans - produced for ECD, Land Restitution, Grade R, CRDP, RECAP and Business Process Services.  First 6 month report for ECD, 2 nd due (late).  Apart from the ECD evaluation, the next evaluation was only commissioned in October 2012 – ie 18 months prior to this report. It is therefore early days to see concrete impacts of the evaluations.  Early days, some examples of policy influence can already be seen:  New ECD policy drafted responding to the ECD Diagnostic Review, including the need to target children from conception;  Renewed focus on quality of Grade R rather than just rollout as a result of the Grade R evaluation – included in MTSF.  Renewed focus on nutrition in children resulting from the evaluation of nutrition interventions for children under 5, and a stunting target in the Medium-Term Strategic Framework. The Improvement Plan will take this further.  Revised Business Process Outsourcing Scheme launched by Minister Davies in UK 14

15. The Presidency: Department of Performance Monitoring and Evaluation Grade R – short-term targets 15

16. The Presidency: Department of Performance Monitoring and Evaluation Key challenges  Inadequate supply of skilled evaluators - range of skills needed from proposal writing, evaluation methodology, writing reports  DPME working with universities and World Bank, promote training of SPs  Insufficient PDI participants  Changing panel criteria to promote access, but keep quality, have 30% PDI requirement in CFPs  Some evaluations are taking a long time to be brought by departments to cluster (often 6+ months) and thence Cabinet.  DPME will bring if takes longer than 4 months.  Departments taking long (6 months+ delay) to finalise improvement plans arising from evaluations, and to report on progress.  Include in MPAT standards and discuss with AG. 16

17. The Presidency: Department of Performance Monitoring and Evaluation Challenges (2)  Weak programme planning  Cabinet agreed need for support programme using government/donor resources. Also develop MPAT standard to drive change.  Insufficient funding for complex evaluations  Programmes should reserve between 0.1% and 5% of programme budgets for evaluation.  Some departments who are funding NEP evaluations themselves are taking a very long time to procure (notably DHS).  These should be in departmental evaluation plans and DPME procures for all evaluations selected for NEP. Need to meet Minister DHS to discuss delays on 5 evaluations.  Some sectors/outcomes not proposing evaluations – notably Health, Economic Infrastructure, Environment, Public Service, Local Government.  FOSAD suggested propose and 6 evaluations being proposed to Cabinet on Crime, Health, Public Service 17

18. The Presidency: Department of Performance Monitoring and Evaluation Taking the evaluation system forward  Undertaking audit of implementation programmes in gov. – may well be 1000! If each were to be evaluated every 5 years = 200/year – big challenge  To do that extending national evaluation system to provinces in 2014-15/2015-16 and to departments from 2015-16. Is an issue of capacity to support 46 national depts and 9 provinces.  Focus on programme planning (again needs resources) 18

19. The Presidency: Department of Performance Monitoring and Evaluation Research  Work starting on a diagnostic which will inform a research strategy – balance between work we do on actual research, vs support to influence the research system so it provides research relevant to the outcomes  Some work being undertaken internally in DPME  PSPPD supporting a research exercise in a number of provinces and departments to see what is happening and what role we should play  Working to develop immediately a repository of key research around each outcome  Assignment on DG workload – starting 19

20. The Presidency: Department of Performance Monitoring and Evaluation  Thank you  Dr Ian Goldman  Head: Evaluation and Research  ian@po-dpme.gov.za ian@po-dpme.gov.za  Www.thepresidency-dpme.gov.za Www.thepresidency-dpme.gov.za20