Policy Lessons Learned Evaluation Or Here we go again…………. Alex Foulds

“One of the great mistakes is to judge policies and programs by their intentions rather than their results” Milton Friedman

It is evaluation that helps us distinguish between intentions and results.

Evaluation ...is assessing, as systematically and objectively as possible, a completed project or programme using data and information that inform strategic decisions, thus improving the project or programme in the future. We draw conclusions around: relevance effectiveness efficiency impact sustainability

Monitoring ...is the systematic and routine collection of information from projects and programmes for four main purposes: To learn from experiences, improve practices and activities in the future; To have internal and external accountability; To inform future decisions;

Why? Recognition that evaluation of programs is a vanishing art To build internal skills within the Department To provide assurance that programs are achieving their objectives How do we react?

Usual Methods - Considerations ANAO BITRE Internal Consultants Hybrid – internal and consultant support Skills development- where does expertise lie Purpose – legislative requirement; internal requirement; financial requirement Resourcing and complexity Timeframes

External/Mixed Evaluation Clear demarcation for hybrid evaluations Previous experience with Department/in the field Clear and detailed schedule of requirement and timeframes Data Inception and regular meetings and review of product What will it look like Who is the audience Badging

Challenges Program purpose Method choice Acceptance Language Politics Resourcing

What is really necessary Commitment Leadership Day job remains Skills and ability Collaboration on method Cross fertilisation – within Department, Consultant All should be better off and better at evaluation

Lessons Learned

Lessons Learned Importance of Program Design Monitoring and Evaluation built in Data collection built in Reality checks You are never the first to do this – you won’t be the last and others are available to assist “First get your facts; then you can distort them at your leisure” Mark Twain

Lessons learned Just because the program is ongoing doesn’t mean evaluation isn’t valuable Evaluation is not an audit – though an audit can be an evaluation Attitude is more important than skillsets Working with colleagues across the department / government / consultants has its own benefits

Lessons Learned Don’t fight it – it won’t go away See it as an opportunity Make sure you get what you asked for We do forget why Ministers want different things for different reasons from the same program In the end though - Is the program effective It’s not just about money

Final Thoughts Need is clear Timing is everything Cooperation is essential Clear objectives and negotiation required PRIOR Evaluation and M&E go together Engage stakeholders early, especially those who are external – believe it or not they probably want to help Define your scope/Terms of Reference document well, especially if using external providers – EARS has templates for Evaluation Plan/scope documents and are happy to assist with the crafting of ToR.  Consider methodology in your plan, including the use of surveys/case studies etc. The SE Roads Branch provides a good example of an area that has positively evolved through the conduct of two priority evaluations. Relevant staff were “reluctant participants” in the HVSPP priority evaluation, and there was a lack of available data to evaluate the program’s performance. However, at the conclusion of the HVSPP evaluation, the team self-nominated R2R for priority evaluation. They were actively involved in all stages of the process, were guided and supported by strong SES leadership, and reduced the time burden on staff by engaging an external supplier to undertake most aspects of the evaluation. In addition, the team requested up-front that a monitoring and evaluation strategy be developed as part of the evaluation, in collaboration with key stakeholders. Involvement in this process has ensured that staff now recognise the benefits of having a strategy in place and the rationale and process for doing so.

Any Questions?