Ian Goldman Head: Evaluation and Research AEA, 17 October 2012 The Presidency Department of Performance Monitoring and Evaluation Using evaluation to influence policy and implementation and accountability: The case of South Africa Ian Goldman Head: Evaluation and Research AEA, 17 October 2012

Outline Background – emergence of evaluation system Key findings and side effects Where next

Political context for M&E in SA 1 Semi-federal system 2 New administration in 2009 focuses on M&E SA is a semifederal state with 9 provinces and around 280 local governments, all autonomous spheres. This means that implementation of policy requires cooperation across spheres, including for M&E As part of the public service reform process, some attention was paid to developing M&E between 1994 and 2009. However, there was a major increase in emphasis on M&E after 2009, as the new administration saw M&E as one way of improving the performance of the public service As a result of the increased emphasis on performance and M&E from 2009, the Department of Performance Monitoring and Evaluation was created in the Presidency in 2010. It has an annual budget of approximately 20 million dollars and a staff complement of approximately 200 people. It is part of the executive 3 DPME established in 2010

Evaluations not being used Evaluation not informing planning, policy-making and budgeting sufficiently, so we are missing the opportunity to improve Government’s effectiveness, efficiency, impact and sustainability.

National Evaluation Plan July 2011 – study tour to Mexico, Colombia, US November 2011 - National Evaluation Policy Framework approved by Cabinet June 2012 First evaluation completes, First National Evaluation Plan with 8 evaluations approved November 2012 - Second National Evaluation Plan approved with 15 evaluations Currently - 3 evaluations completed, 4 close to completion, 16 others underway, next 15 for 2014/15 selected

System design Evaluation and Research Unit in DPME/Presidency National Evaluation Plan Evaluation and Research Unit in DPME/Presidency Policy Framework setting out common approach/language Focus on utilisation Maximise ownership so evaluations proposed by depts Cost sharing by dept/DPME Improvement plans, evaluations public Focus on strategic priorities through 3 year National and Provincial Evaluation plans National Evaluation System Support by cross-Government Evaluation Technical Working Group Use of Steering Committee as semi-independent bodies to make decisions, with DPME Guidelines, standards, training, peer reviewers, quality assurance, evaluation repository etc - minimum standards Widen use through Provincial/Departmental Evaluation Plans

Early days but key emerging results First evaluations are being influential Early Childhood Development (ECD) evaluation changing policy and resources for services Grade R leading to rethinking of strategies Business Process Services leading to tweaking implementation Change President, Parliament and party structures anxious to get results Pressure Need to improve planning system (Programmes, Diagnostics prior) Logframe Expenditure reviews And better understand budgets

Key findings - side effects Some fear leading to gaming strategies: Fear that evaluations public Running parallel internal evaluation Seeking to influence sample Seeking to delay findings being made public Senior managers keeping away Challenge of adequate supply of good quality evaluators Concern at use of consultants

Way forward This year is critical – monitor and document: How to deal with gaming – link to management process How government clusters/Cabinet deal with difficult results (and elections are next year) Response from Portfolio Committees Some elements planned Demand - training in analysis and use of evidence for Directors-General/Deputies as well as Parliament Demand – support provinces/depts Supply of evaluators – interact with universities Attitude - encourage evaluators to express constructively Planning - training around programme planning Budget - project on expenditure reviews Learning - bring stakeholders together to reflect on learnings

Contact Dr Ian Goldman Head: Evaluation and Research Department of Performance M&E, The Presidency ian@po-dpme.gov.za Twitter @iangoldmansa www.po-dpme.gov.za

Additional slides

Implementation evaluation Different types of evaluations related to questions around the outcome model Impact evaluation Has the intervention had impact at outcome and impact level, and why Economic Evaluation What are the cost-benefits? Implementation evaluation - what is happening and why If presenting the following slides on each type of evaluation, then can be brief here. Mention evaluation synthesis too. DESIGN Diagnostic what is the underlying situation and root causes of the problem, what are options? Design evaluation Does the theory of change seem strong?