1. Policy Lessons Learned
Evaluation Or
Here we go again………….
Alex Foulds

2. “One of the great mistakes is to judge policies and programs by their intentions rather than their results”
Milton Friedman

3. It is evaluation that helps us distinguish between intentions and results.

4. Evaluation
...is assessing, as systematically and objectively as possible, a completed project or programme using data and information that inform strategic decisions, thus improving the project or programme in the future. We draw conclusions around:
relevance
effectiveness
efficiency
impact
sustainability

5. Monitoring
...is the systematic and routine collection of information from projects and programmes for four main purposes:
To learn from experiences, improve practices and activities in the future;
To have internal and external accountability;
To inform future decisions;

6. Why?
Recognition that evaluation of programs is a vanishing art
To build internal skills within the Department
To provide assurance that programs are achieving their objectives
How do we react?

8. Usual Methods - Considerations
ANAO
BITRE
Internal
Consultants
Hybrid – internal and consultant support
Skills development- where does expertise lie
Purpose – legislative requirement; internal requirement; financial requirement
Resourcing and complexity
Timeframes

9. External/Mixed Evaluation
Clear demarcation for hybrid evaluations
Previous experience with Department/in the field
Clear and detailed schedule of requirement and timeframes
Data
Inception and regular meetings and review of product
What will it look like
Who is the audience
Badging

10. Challenges Program purpose Method choice Acceptance Language Politics
Resourcing

12. What is really necessary
Commitment
Leadership
Day job remains
Skills and ability
Collaboration on method
Cross fertilisation – within Department, Consultant
All should be better off and better at evaluation

13. Lessons Learned

14. Lessons Learned Importance of Program Design
Monitoring and Evaluation built in
Data collection built in
Reality checks
You are never the first to do this – you won’t be the last and others are available to assist
“First get your facts; then you can distort them at your leisure”
Mark Twain

15. Lessons learned
Just because the program is ongoing doesn’t mean evaluation isn’t valuable
Evaluation is not an audit – though an audit can be an evaluation
Attitude is more important than skillsets
Working with colleagues across the department / government / consultants has its own benefits

16. Lessons Learned Don’t fight it – it won’t go away
See it as an opportunity
Make sure you get what you asked for
We do forget why
Ministers want different things for different reasons from the same program
In the end though - Is the program effective
It’s not just about money

17. Final Thoughts Need is clear Timing is everything
Cooperation is essential
Clear objectives and negotiation required PRIOR
Evaluation and M&E go together
Engage stakeholders early, especially those who are external – believe it or not they probably want to help
Define your scope/Terms of Reference document well, especially if using external providers – EARS has templates for Evaluation Plan/scope documents and are happy to assist with the crafting of ToR.  Consider methodology in your plan, including the use of surveys/case studies etc.
The SE Roads Branch provides a good example of an area that has positively evolved through the conduct of two priority evaluations. Relevant staff were “reluctant participants” in the HVSPP priority evaluation, and there was a lack of available data to evaluate the program’s performance.
However, at the conclusion of the HVSPP evaluation, the team self-nominated R2R for priority evaluation. They were actively involved in all stages of the process, were guided and supported by strong SES leadership, and reduced the time burden on staff by engaging an external supplier to undertake most aspects of the evaluation.
In addition, the team requested up-front that a monitoring and evaluation strategy be developed as part of the evaluation, in collaboration with key stakeholders. Involvement in this process has ensured that staff now recognise the benefits of having a strategy in place and the rationale and process for doing so.

19. Any Questions?