1. Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector Bronte Koop, Principal Consultant, Strategy and Evaluation John Collins, Manager, Program Evaluation

2. COAG Reform Agenda: Human Capital Development National Partnerships Literacy and Numeracy Teacher Quality Low SES communities

3. Not getting what you expected from an evaluation

4. What do we mean by Snake Oil Staff as passive recipients Evaluation questions not addressed Use lots of words Recommendations to continue the program and evaluate it again Lack of evidence based recommendations for improvement No discussion of data or design limitations No discussion of alternative explanations for program effect Evaluators reluctant to address client concerns about clarity or focus

6. Useful information to guide decision making

7. So what’s the Good Oil Staff critical participants Provides answers to specific questions Balances qualitative with quantitative data Reports are just sufficient in length Conclusions follow from the evidence Has logical recommendations Data or design limitations are discussed Alternative explanations for program effect are discussed

8. How do we get there? teamwork commitment learning proactivity

9. What are we trying to change? Attitudes and values Knowledge and skills Organisational behaviour

10. Overcoming your fears is the first step

11. Principles for evidence based program management Demand evidence Examine the logic Treat the program as an unfinished prototype Embrace an attitude of wisdom Jeffrey Pfeffer and Robert Sutton on evidence based management

12. 1.Executive leadership 2.Consulting Services 3.Resources 4.Professional development 5.Policy and operations 6.Networks 7.Strategic Oversight Where should be put our efforts? CREATING CHANGE

13. Locate near the top Recruit the knowledge, skills and motivation Build credibility Provide an ‘evaluation perspective’ Consulting Services

14. Resources If you build it, they will come

15. What have others done?

16. Accessible Resources

17. Practical Resources

18. Provide a starting point

19. Project Management

20. Evaluation & planning

21. Professional Development Time to stretch your thinking Induction Customized training to needs

22. Procurement: Expression of Interest Two part contracts: plan + do Executive summaries Interviews Hard questions Policy and Operations Think outside the square

23. Network, network, network

24. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

25. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

26. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes...

27. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes... current investments will be evaluated and we will reinvest where appropriate.

28. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes... current investments will be evaluated and we will reinvest where appropriate.

30. Represents a major cultural change.

31. Represents a major cultural change From an ‘accountability’ model to one centred on the corporate goal.

32. Represents a major cultural change From an ‘accountability’ model to one centred on the corporate goal.

33. How?

34. A marketing approach

35. A marketing approach Product life-cycle

36. A marketing approach Product life-cycle

37. A marketing approach Product life-cycle Customer

38. A marketing approach Product life-cycle Customer –Needs and wants

39. A marketing approach Product life-cycle Customer –Needs and wants

40. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

41. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

42. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

43. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

44. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity

45. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity

46. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition

47. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition

48. A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition 4 Ps – Profit Product Place Promotion

49. Priorities? To continue development & refinement of our ‘system’ To address changing the focus of our evaluation culture To move from ‘good oil’ to ‘best value’ oil

50. Disclaimer The views and interpretations expressed in this presentation are those of the authors and do not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia

51. Contact Bronte.Koop@sa.gov.au John.Collins2@sa.gov.au