Department of Performance Monitoring and Evaluation The Presidency Department of Performance Monitoring and Evaluation Developments in Government: National Evaluation System and broader context Dr Sean Phillips Director General Presentation to SAMEA 19 September2013

Focus of DPME to date M&E of national priorities Management performance M&E M&E of front-line service delivery Government-Wide M&E System Plans for the 12 priority outcomes (delivery agreements) Monitoring (ie tracking) progress against the plans Evaluating to see how to improve programmes, policies, plans (2012-13 8 evaluations, then 15 per year) Monitoring of experience of citizens when obtaining services (joint with provinces), including citizen-based monitoring Presidential Hotline – analysing responses and follow-up Assessing quality of management practices in individual departments (MPAT) at all levels of government Moderated self assessment and continuous improvement M&E platforms across gov – nationally, provincially Data quality/Structures of M&E units/Capacity development Emerging focus on (implementation) programmes National Evaluation System National M&E Policy Framework To date my Department has been focused on three levels of performance monitoring and evaluation. At the highest level, our focus since 2010 has been on facilitating the development of plans for priorities such as basic education, health, reducing crime and creating employment. The main aims of this initiative have been to increase the strategic focus of government, to introduce results-based planning on a sectoral basis, to increase coordination between departments and across spheres of government, and to use M&E of progress against the plans to foster a culture of evidence-based continuous improvement. The 10 and 15 year reviews of the post-apartheid government’s performance which were carried out by the Presidency came to the conclusion that a key challenge in SA is implementation of policies, which in turn is related to management weaknesses. At the next level we have therefore introduced a management performance assessment mechanism, informed by the Canadian Management Assessment Framework. The methodology is based on self assessment against standards in key management areas, coupled with verification against secondary data. It is being implemented in partnership with the provinces and the results of the assessments will be presented to Cabinet and the Provincial Executive Councils on an ongoing basis, together with monitoring reports on the implementation of improvement plans. A key political imperative in South Africa is to improve the quality of services provided directly to citizens. We are therefore also carrying out M&E at the level of frontline service delivery by visiting service delivery sites together with the Offices of the Premier in the provinces. The focus is on whether service delivery standards are in place and are being adhered to. The results and monitoring reports on the implementation of improvement plans are similarly presented to Cabinet and the Provincial Executive Councils. We have also introduced a Presidential Hotline through which the public can lodge service delivery complaints. In future we would like to introduce more citizen-based monitoring, in partnership with civil society organisations.

Timelines 2010 DPME established. 12 outcomes agreed, Minister’s performance agreements, delivery agreements for each outcome, quarterly reports start, using traffic light system 2011 Systems for Management Performance Assessment created, monitoring of front-line services, and Evaluation Policy Framework. MPAT assessment of 103/156 national and provincial departments 2012 Reports against all of these, first evaluation completed. First National Evaluation Plan (NEP) approved for 2012/13 with 8 evaluations. Second NEP approved for 2013/14 to 2015/16 with 15 evaluations in 2013/14. 8 evaluations start for 2012/13 MPAT assessment of all 156 national and provincial departments 2013 15 evaluations underway or being scoped for 2013/14 Work with Treasury to develop system for performance dialogue Systems for monitoring local government and citizen-based monitoring being piloted Government-wide M&E Policy Framework drafted

Why evaluate? Improving policy or programme performance (evaluation for continuous improvement): this aims to provide feedback to programme managers. Evaluation for improving accountability: where is public spending going? Is this spending making a difference? Improving decision-making: Should the intervention be continued? Should how it is implemented be changed? Should increased budget be allocated? Note Minister clear he wants all 4 Evaluation for generating knowledge (for learning): increasing knowledge about what works and what does not with regards to a public policy, programme, function or organization.

Key aspects of the SA national evaluation system Departments submit proposals for interventions to evaluate (policies, programmes, projects) – as they have to own the evaluation and implement the findings. 3rd parties, eg Treasury, Parliament can propose evaluations, but departments should normally submit Selection by cross-government Evaluation Technical Working Group – based on importance (either by scale or because strategic or innovative) Evaluations must be made public unless security concerns All evaluation reports in the national system go to Cabinet (which approves the Plan) To ensure independence: Evaluations implemented as partnership between department(s) and DPME Steering Committee makes decisions on evaluation not department External service providers undertake the evaluation reporting to the Steering Committee To ensure quality: Peer reviewers (normally 2) per evaluation Evaluation panel, standards, guidelines, training etc Quality assessment once completed Joint funded – department and DPME, in some cases donors There must be an improvement plan which is monitored

Implementation evaluation Different types of evaluations related to questions around the outcome model Economic Evaluation What are the cost-benefits? Impact evaluation Has the intervention had impact at outcome and impact level, and why Implementation evaluation - what is happening and why If presenting the following slides on each type of evaluation, then can be brief here. Mention evaluation synthesis too. DESIGN Diagnostic what is the underlying situation and root causes of the problem Design evaluation Does the theory of change seem strong?

Progress with Evaluations National Evaluation Plan : 2012/13: 7 evaluations 2013/14: 15 evaluations 2014/15: 16 evaluations selected but not yet approved by Cabinet Evaluations completed - ECD completed June 2012 (on DPME website), Grade R and Business Process Outsourcing reports completed and on way to Cabinet 17 other evaluations underway or being scoped from 2012/13 and 2013/14 Audit of evaluations from 2006 identified 83 evaluations , quality assessment tool developed and applied, 71 passed and are up on the DPME website, and assessment contributed to the awards for best evaluation yesterday. Evaluation repository launched yesterday

Progress with National Evaluation System >12 Guidelines and templates - ranging from Guideline on TORs to Guideline on Improvement Plans Standards for evaluations and competences drafted, and standards have guided a quality assessment tool 2 courses developed, over 250 government staff trained Evaluation panel developed with 42 organisations which simplifies procurement – however evaluation capacity is still limited – and a particular problem that few universities are bidding for evaluations Gauteng, W Cape provinces have developed provincial evaluation plans. DPME is working with other provinces who wish to develop PEPs, starting with Free State 3 departments have developed a departmental evaluation plan (dti, DST, DRDLR)

Evaluation challenges emerging Overall the system is working well but some challenges. These include: Poor programme plans (for the government programmes which are being implemented) and so difficult to evaluate - need for minimum standards for programme plans – DPME has issued guideline on this Poor communication channels - programme managers often not aware of the possibility of conducting evaluations on their programmes Some senior managers wary of evaluation and don’t see it as an opportunity to improve their performance Making sure evaluations proposed are strategic ones and that key sectors covered Sometimes departments not budgeting for evaluations and expecting DPME to provide all the money Departments not planning ahead – very important for impact evaluations in particular where need to plan 3+ years ahead, also affects how rollout happens Reluctance to rollout in carefully planned way which facilitates impact evaluation. To be clear on impact must compare with/without the intervention

Conclusions Interest is growing – more departments getting involved, more provinces, and more types of evaluation Some 35 evaluations are underway or planned Development of guideline for planning implementation programmes and for depts to do design evaluation will potentially have very big impact – will build capacity in departments to undertake these Challenges emerging as the evaluation reports start being finalised and raise challenges which some departments are sensitive about results Parliament is playing a key oversight role in this regard – committees are starting to request departments to present the evaluation results to them